Validation of cross logical groups in a network

ABSTRACT

Disclosed are systems, methods, and computer-readable media for assuring tenant forwarding in a network environment. Network assurance can be determined in layer 1, layer 2 and layer 3 of the networked environment including, internal-internal (e.g., inter-fabric) forwarding and internal-external (e.g., outside the fabric) forwarding in the networked environment. The network assurance can be performed using logical configurations, software configurations and/or hardware configurations.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/788,616, filed on Feb. 12, 2020, which in turn, is a continuation ofU.S. patent application Ser. No. 15/663,638, now U.S. Pat. No.10,567,288, granted Feb. 18, 2020, which claims the benefit of U.S.Provisional Patent Application Ser. No. 62/521,774 filed on Jun. 19,2017, the contents of which are incorporated by reference in theirentireties.

TECHNICAL FIELD

The present technology pertains to network configuration and assurance,and more specifically to routing and forwarding configuration andassurance internal and external to a network.

BACKGROUND

Computer networks are becoming increasingly complex, often involving lowlevel as well as high level configurations at various layers of thenetwork. For example, computer networks generally include numerousaccess policies, forwarding policies, routing policies, securitypolicies, quality-of-service (QoS) policies, etc., which together definethe overall behavior and operation of the network. Network operatorshave a wide array of configuration options for tailoring the network tothe needs of the users. While the different configuration optionsavailable provide network operators a great degree of flexibility andcontrol over the network, they also add to the complexity of thenetwork. In many cases, the configuration process can become highlycomplex. Not surprisingly, the network configuration process isincreasingly error prone. In addition, troubleshooting errors in ahighly complex network can be extremely difficult. The process ofidentifying the root cause of undesired behavior in the network can be adaunting task.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which the above-recited and otheradvantages and features of the disclosure can be obtained, a moreparticular description of the principles briefly described above will berendered by reference to specific embodiments thereof which areillustrated in the appended drawings. Understanding that these drawingsdepict only exemplary embodiments of the disclosure and are nottherefore to be considered to be limiting of its scope, the principlesherein are described and explained with additional specificity anddetail through the use of the accompanying drawings in which:

FIGS. 1A and 1B illustrate example network environments;

FIG. 2A illustrates an example object model for a network;

FIG. 2B illustrates an example object model for a tenant object in theexample object model from FIG. 2A;

FIG. 2C illustrates an example association of various objects in theexample object model from FIG. 2A;

FIG. 2D illustrates a schematic diagram of example models forimplementing the example object model from FIG. 2A;

FIG. 3A illustrates an example network assurance appliance;

FIG. 3B illustrates an example system for network assurance;

FIG. 3C illustrates a schematic diagram of an example system for networkassurance.

FIG. 3D illustrates an example of validated policies;

FIG. 4 illustrates an example platform for distributed fault codeaggregation;

FIGS. 5A and 5B illustrate example method embodiments for networkassurance and fault code aggregation;

FIG. 6 illustrates an example checker of the network assuranceappliance;

FIG. 7 illustrates an example network environment;

FIG. 8 illustrates an example method embodiment for network assurance ofLayer 1 interfaces;

FIG. 9 illustrates an example method embodiment for network assurance ofLayer 2 networking;

FIG. 10 illustrates an example method embodiment for network assuranceof Layer 3 networking;

FIG. 11 illustrates an example configurations for network assurance;

FIG. 12 illustrates an example network environment and configuration fornetwork assurance of Cross Logical Groups/VRFs;

FIG. 13 illustrates an example network environment and configuration fornetwork assurance of the RIB and FIB;

FIG. 14 illustrates an example method embodiment for network assuranceof the RIB and FIB;

FIG. 15 illustrates an example method embodiment for network assuranceof Layer 3 out;

FIG. 16 illustrates an example network diagram for BD-L3out association;

FIG. 17 illustrates an example method embodiment for network assuranceof BD-L3out association;

FIG. 18 illustrates an example method embodiment for network assuranceof Learned Routes;

FIG. 19 illustrates an example diagram of subnet overlaps;

FIG. 20 illustrates an example method embodiment for network assuranceof overlapping subnets;

FIG. 21 illustrates an example network device in accordance with variousembodiments; and

FIG. 22 illustrates an example computing device in accordance withvarious embodiments.

DESCRIPTION OF EXAMPLE EMBODIMENTS

Various embodiments of the disclosure are discussed in detail below.While specific implementations are discussed, it should be understoodthat this is done for illustration purposes only. A person skilled inthe relevant art will recognize that other components and configurationsmay be used without parting from the spirit and scope of the disclosure.Thus, the following description and drawings are illustrative and arenot to be construed as limiting. Numerous specific details are describedto provide a thorough understanding of the disclosure. However, incertain instances, well-known or conventional details are not describedin order to avoid obscuring the description. References to one or anembodiment in the present disclosure can be references to the sameembodiment or any embodiment; and, such references mean at least one ofthe embodiments.

Reference to “one embodiment” or “an embodiment” means that a particularfeature, structure, or characteristic described in connection with theembodiment is included in at least one embodiment of the disclosure. Theappearances of the phrase “in one embodiment” in various places in thespecification are not necessarily all referring to the same embodiment,nor are separate or alternative embodiments mutually exclusive of otherembodiments. Moreover, various features are described which may beexhibited by some embodiments and not by others.

The terms used in this specification generally have their ordinarymeanings in the art, within the context of the disclosure, and in thespecific context where each term is used. Alternative language andsynonyms may be used for any one or more of the terms discussed herein,and no special significance should be placed upon whether or not a termis elaborated or discussed herein. In some cases, synonyms for certainterms are provided. A recital of one or more synonyms does not excludethe use of other synonyms. The use of examples anywhere in thisspecification including examples of any terms discussed herein isillustrative only, and is not intended to further limit the scope andmeaning of the disclosure or of any example term. Likewise, thedisclosure is not limited to various embodiments given in thisspecification.

Without intent to limit the scope of the disclosure, examples ofinstruments, apparatus, methods and their related results according tothe embodiments of the present disclosure are given below. Note thattitles or subtitles may be used in the examples for convenience of areader, which in no way should limit the scope of the disclosure. Unlessotherwise defined, technical and scientific terms used herein have themeaning as commonly understood by one of ordinary skill in the art towhich this disclosure pertains. In the case of conflict, the presentdocument, including definitions will control.

Additional features and advantages of the disclosure will be set forthin the description which follows, and in part will be obvious from thedescription, or can be learned by practice of the herein disclosedprinciples. The features and advantages of the disclosure can berealized and obtained by means of the instruments and combinationsparticularly pointed out in the appended claims. These and otherfeatures of the disclosure will become more fully apparent from thefollowing description and appended claims, or can be learned by thepractice of the principles set forth herein.

Overview

Disclosed are systems, methods and non-transitory computer readablemedium (CRM) for network assurance of a network. The systems, methodsand non-transitory computer readable medium are configured to receive,from a controller, a global logical model in a first format, the globallogical model containing instructions on how endpoints connected to anetwork fabric communicate within the fabric and receive, from one ormore network devices within the fabric, a software model being at leasta subset of instructions from the global logical model in a secondformat executable on the one or more network devices, the subset ofinstructions being instructions from the global logical model that arespecific to operability of the one or more network devices. The systems,methods and non-transitory computer readable medium are also configuredto create a local logical model in the first format, the local logicalmodel being at least a portion of the received global logical modelcontaining instructions on how endpoints connected to a network fabriccommunicate within the fabric, convert at least a portion of the createdlocal logical model and at least a portion of the received softwaremodel into a common format, and compare content of overlapping fieldsfrom the common format of the local logical model and the common formatof the software model, wherein a positive outcome of the comparisonrepresents that the one or more network devices accurately created thereceived software model from the received global logical model. In someexamples, the first and second formats are different from each other,the common format is one of the first and second formats, or isdifferent from both of the first and second formats.

In some examples, the systems, methods and CRM can be configured toreceive from the one or more network devices within the fabric, ahardware model of hardware configurations converted from the softwaremodel. The systems, methods and CRM can be configured to convert atleast a portion of the received software model and at least a portion ofthe received hardware model into the common format and compare contentof overlapping fields from the common format of the software model withthe common format of the hardware model, and a positive outcome of thefirst and second comparisons represents that that the one or morenetwork devices accurately created the hardware model from the globallogical model. In some examples, the hardware model has a format that isdifferent from the first format and the second format, the second formatis one of the second and third formats, or is different from both of thesecond and third formats.

Also disclosed are systems, methods and CRM for event generation innetwork assurance of a network. The systems, methods and CRM can beconfigured to receive from a controller a global logical model in afirst format, the global logical model containing instructions on howendpoints connected to a network fabric communicate within each otherthrough one or more network devices within the fabric and receive fromone or more network devices within the fabric a software model being atleast a subset of instructions from the global logical model in a secondformat executable on the one or more network devices, the subset ofinstructions being instructions from the global logical model that arespecific to operability of the one or more network devices and ahardware model of hardware configurations converted from the softwaremodel. The systems, methods and CRM can also be configured to validateaccuracy of the received global logical model, the received softwaremodel and the received hardware model and generate one or more eventsbased on the validating

In some examples, systems, methods and CRM can be configured to storeprior configurations and corresponding effectiveness of the priorconfigurations, identify any configuration in the received globallogical model, the received software model and/or the receiving hardwaremodel that is the same or similar to a stored prior configuration with acorresponding adverse effectiveness, and report of potential flaws inthe received global logical model, the received software model and/orthe receiving hardware model in response to a positive result of theidentifying.

In some examples, the one or more events include an informational eventin response to no identified inconsistencies resulting from thevalidating. In other examples, the one or more events includes an errorevent in response to at least one identified inconsistency resultingfrom the validating. In some examples, the error event can havedifferent levels of severity based on a severity of the at least oneinconsistency. In some examples, in response to an error event, thesystem further comprising instructions, which when executed by the atleast one processor causes the at least one processor to independentlyvalidate secondary content within the received global logical model, thereceived software model and/or the receiving hardware model, thesecondary content being related to original content for which the atleast one inconsistency was generated.

Also disclosed are systems, methods and CRM for performing a Layer 1network assurance check of proper deployment of a configuration in afabric. The systems, methods and CRM can be configured to receive from acontroller, a global logical model containing instructions on howendpoints connected to a network fabric communicate within the fabric,the instructions including access policies for at least one interfaceand receive from one or more network devices within the fabric, asoftware model being at least a subset of instructions from the globallogical model, the subset of instructions being instructions from theglobal logical model that are specific to operability of the one or morenetwork devices. In some examples, the systems, methods and CRM can beconfigured to validate that at least Layer 1 of the access policieswithin the received global logical model are properly configured on theone network devices, identify within the received software model areported state of a physical link and a software link of one or moreports of the one or more network devices, obtain an actual state of thephysical link and the software link of the one or more ports of the oneor more network devices, compare the reported state of the physical linkand the software link with the obtained actual state of the physicallink and the software link and generate one or more events based on thevalidation and/or the comparison.

In some examples, systems, methods and CRM can be configured to confirmthe received global logical model, a switch profile and an interfaceprofile are present, the switch profile and interface profile areproperly linked, and that the interface profile is properly linked to aglobal profile.

In some examples, systems, methods and CRM can be configured to generatean error event in response to the determination that one or more of theaccess policies are not configured on the at least one interface.

In some examples, systems, methods and CRM can be configured to generatean error even in response to the comparison indicating that the reportedstate of the physical link is active and the actual state of thephysical link is down, or the reported state of physical link is downand the actual state of the physical link is active.

In some examples, systems, methods and CRM can be configured to generatean error event in response to the comparison indicating that thereported state of the software link is active and the actual state ofthe physical switch is down, or the reported state of physical switch isdown and the actual state of the physical switch is active. In someexamples, systems, methods and CRM can be configured to poll the one ormore network devices for the actual state of the physical link and thesoftware link.

Also disclosed are systems, methods and CRM for performing a Layer 2network assurance check of proper deployment of a configuration in afabric. The systems, methods and CRM can be configured to receive, froma controller, a global logical model in a first format, the globallogical model containing instructions on how endpoints connected to anetwork fabric communicate within the fabric and receive, from one ormore network devices within the fabric, a software model being at leasta subset of instructions from the global logical model in a secondformat executable on the one or more network devices, the subset ofinstructions being instructions from the global logical model that arespecific to operability of the one or more network devices. The systems,methods and CRM can be configured to create a local logical model in thefirst format, the local logical model being at least a portion of thereceived global logical model that is specific to operability of the oneor more network devices, convert at least a portion of Layer 2 contentof the created local logical model and at least a portion of Layer 2content of the received software model into a common format and comparecontent of at least some Layer 2 overlapping fields from the commonformat of the created local logical model and the common format of thereceived software model. In some examples, a positive outcome of thecomparison represents that the one or more network devices earlier atleast partially accurately created the software model from the globallogical model. The systems, methods and CRM can also be configured tovalidate that at least some Layer 2 content of the global logical modelis properly configured.

In some examples, at least a portion of Layer 2 content subject to theconversion and the comparison includes VLAN information and interfaceinformation, to thereby at least partially check correct deployment ofany bridge domains and/or endpoint groups (EPG). In some examples theleast some of the Layer 2 content subject to the validation includes, atLayer 2, that access policies are configured properly, EPGs areconfigured properly, that multiple EPGs when present on the one of thenetwork devices are not using the same virtual local area network(VLAN), there are no overlapping VLANs, no duplicate VLANs at the switchlevel on a same switch, and no duplicate VLANS on a port level on thesame port of a same switch.

The systems, methods and CRM can also be configured to receive from theone of the one or more network devices within the fabric a hardwaremodel in a third format of hardware configurations converted from thesoftware model, convert at least a portion of Layer 2 content of thereceived software model and/or at least a portion of Layer 2 content ofthe hardware model into the common format. and compare content of atleast some of Layer 2 overlapping fields from the common format of thereceived software model with the common format of the received hardwaremodel. In some examples, a positive outcome of the comparison representsthat the one or more network devices at least partially accuratelyconverted Layer 2 of the software model into the hardware model. In someexamples, at least a portion of the Layer 2 content subject to theconversions and the comparisons includes the VLAN for each EPG.

The systems, methods and CRM can also be configured receive aDistributed Virtual Switch (DVS) logical model from a DVS outside of thefabric, convert at least a portion of Layer 2 content of the receivedDVS logical model and/or at least a portion of Layer 2 content of thereceived global hardware model into a third common format and comparecontent of at least some of Layer 2 overlapping fields from the thirdcommon format of the received global logical model with the third commonformat of the received DVS logical model. In some examples, a positiveoutcome of the third comparing represents VLANs within the globallogical model have been properly assigned across the network.

Also disclosed are systems, methods and CRM for performing a Layer 3 BDsubnets properly deployed network assurance check of proper deploymentof a configuration in a fabric. The systems, methods and CRM can beconfigured to receive, from a controller, a global logical model in afirst format, the global logical model containing instructions on howendpoints connected to a network fabric communicate within the fabric,the global logical model including at least one virtual routing andforwarding instance (VRF) and receive, from one or more network deviceswithin the fabric, a software model being at least a subset ofinstructions from the global logical model in a second format executableon the one or more network devices, the subset of instructions beinginstructions from the global logical model that are specific tooperability of the one or more network devices. The systems, methods andCRM can also be configured to convert, for each network device, theglobal model into a local logical model in the first format, the locallogical model being at least a portion of the received global logicalmodel that is specific to operability of the corresponding each networkdevice, create a container for each VRF in the received global logicalmodel, populate each of the created containers with the local logicalmodel and the software model for each of the network devices associatedwith the VRF and confirm bridge domain (BD) subnets in the populatedcontainers match. In some examples, the population includes a set union,such that the populated VRF container does not contain any duplicativeBD subnets.

In some examples, the populated container includes a software modelwithout a corresponding local logical model represents an improper extraitem. In some examples, the populated container includes a local logicalmodel without a corresponding software model represents an error indeployment of the global logical model.

The systems, methods and CRM can also be configured to subtract one ofthe software models from a corresponding local logical hardware model,subtract the corresponding local logical model from the one of thesoftware models. In some examples, a mismatch between the subtractionsrepresents a discrepancy. In some examples, an error event is generatedin response to a mismatch between the subtractions. In some examples, anerror event is generated in response to a mismatch in the bridge domain(BD) subnets in the populated container.

Also disclosed are systems, methods and CRM for performing a Layer 3 VRFcontainer network assurance check of proper deployment of aconfiguration in a fabric. The systems, methods and CRM can beconfigured to receive a global logic model, a plurality of softwaremodels and a plurality of hardware models, the global logic modelincluding virtual routing instance (VRF). The systems, methods and CRMcan also be configured to create a plurality of local logical modelsfrom the global logical model, create, for the received VRF, a VRFcontainer. populate the created VRF container with a subset of thereceived software models, the received hardware models, and/or thecreated local logical models, the subset being defined by one or morenetwork devices in the fabric that are associated with the VRF of thereceived global logical model and identify within the populated VRFcontainer one or more of the received software models, the receivedhardware models, and/or the created local logical models that correspondto one or more network devices in the fabric that is not associated withthe VRF of the received global logical model. In some examples, apositive result of the identification represents a discrepancy in theglobal logical model.

In some examples, systems, methods and CRM can be configured to validatethat the global logical model is consistent with the software modelsand/or the hardware models. In some examples, systems, methods and CRMcan be configured to compare overlapping fields of content between oneof the local logic models and one of the software models. In someexamples, systems, methods and CRM can be configured to compareoverlapping fields of content between one of the software models and oneof the hardware models. In some examples, systems, methods and CRM canbe configured to generate an error event in response to a positiveresult of the identification.

In some examples, the software models are based on the global logicalmodel. In some examples, the hardware models are based on correspondingsoftware models.

Also disclosed are systems, methods and CRM for performing a RIB-FIBnetwork assurance check of proper deployment of a configuration in afabric. The systems, methods and CRM can be configured to obtain aforwarding information base (FIB) and a routing information base (RIB)of the network device, convert the FIB and/or the RIB to a commonformat, remove from the RIB and FIB duplicates, determine whether a RIBentry in the RIB matches an entry in the FIB, identify, in response to anegative result of the determination, when the entry in the FIB iscovered by another RIB entry. In some examples, an error event can begenerated in response to a negative result of the identification.

In some examples, systems, methods and CRM can be configured todetermine an IP address/mask and next hop of an entry in the FIB matchesthe RIB entry. In some examples, systems, methods and CRM can beconfigured to identify a FIB entry in the FIB that has a next hop thatmatches the unmatched RIB entry and an IP address/mask that covers theunmatched RIB entry. In some examples, systems, methods and CRM can beconfigured to identify a FIB entry in the FIB that has an alternativesubnet prefix that completely covers the unmatched RIB entry.

In some examples, the FIB is obtained from a line controller of thenetwork device and the RIB is extracted from SUP controller of thenetwork device. In some examples, the network device is a leaf or spinein the fabric, and the system, method and CRM can be configured toobtain from the leaf or spine a software model containing the FIB andRIB. In some examples, systems, methods and CRM can be configured todetermine is applied to every entry in the RIB.

Also disclosed are systems, methods and CRM for performing a Cross EndPoint Group network assurance check of proper deployment of aconfiguration in a fabric. The systems, methods and CRM can beconfigured to receive a global logic model, a plurality of softwaremodels and/or a plurality of hardware models, the global logic modelincluding a virtual routing and forwarding instance (VRF), the VRFhaving under it at least one bridge domain (BD) and at least oneassociated EPG. The systems, methods and CRM can also be configured tocreate a plurality of local logical models from the received globallogical model, create, for the VRF of the received global logical model,a VRF container, populate the created VRF container with a subset of thesoftware models, the hardware models, and/or the local logical models,the subset being defined by leafs in the fabric on which the VRF isdeployed, determine whether a security contract exists between any ofthe at least one EPG in the VRF container and an EPG not in the VRFcontainer of the received global logical model and validate, in responseto a positive result of the determination, that one or more subnets donot clash.

In some examples, each of the at least one BD includes at least onesubnet. In some examples, each EPG of the at least one EPG includes atleast one subnet.

The systems, methods and CRM can also be configured to determine a firstset of subnets in a first BD associated with a first EPG where thecontract exists, determine second set of subnets in a second BDassociated with a second EPG where the contract exists and validate thatthe first set of subnets does not intersect with the second set ofsubnets. The systems, methods and CRM can also be configured to validatefor each of the subnets, a next hop.

In some examples, an error event is generated in response to a clashbetween the one or more subnets. In some examples, an error event isgenerated in response to the first set of subnets intersecting with thesecond set of subnets.

Also disclosed are systems, methods and CRM for performing anoverlapping subnet network assurance check of proper deployment of aconfiguration in a fabric. The systems, methods and CRM can beconfigured to receive, from a controller, a global logical model in afirst format, the global logical model containing instructions on howendpoints connected to a network fabric communicate within the fabricand receive, from one or more network devices within the fabric, asoftware model being at least a subset of instructions from the globallogical model in a second format executable on the one or more networkdevices, the subset of instructions being instructions from the globallogical model that are specific to operability of the one or morenetwork devices. The systems, methods and CRM can also be configured todetermine whether one or more overlapping bridge domain (BD) subnets inthe received global logical model and the received software models, andin response to the determination an overlapping BD subnet of the one ormore overlapping BD subnets, determine whether any of the one or moreoverlapping BD subnets satisfy an exception. In some examples, anegative result of either of the determinations at least partiallyrepresents that subnets have been properly deployed. In some examples,an error event is generated in response to overlap without an applicableexception.

The systems, methods and CRM can be configured to inspect IP addressesand masks of the software models and determine an overlap when two ormore of the IP addresses and masks match. In some examples, the locatingis performed in each VRF of each of the network devices. In someexamples, an exception is when a BD subnet is within a learned route. Insome examples, a positive result of either of the determinations atleast partially represents that subnets have not been properly deployed.In some examples, an exception is when overlapping BD subnets are thesame.

Also disclosed are systems, methods and CRM for performing an L3outnetwork assurance check of proper deployment of a configuration in afabric. The systems, methods and CRM can be configured to receive, froma controller, a global logical model in a first format, the globallogical model containing instructions on how endpoints connected to anetwork fabric communicate within the fabric and receive, from one ormore network devices within the fabric, a software model being at leasta subset of instructions from the global logical model in a secondformat executable on the one or more network devices, the subset ofinstructions being instructions from the global logical model that arespecific to operability of the one or more network devices. The systems,methods and CRM can also be configured to create a local logical modelin the first format, the local logical model being at least a portion ofthe received global logical model that is specific to operability of theone or more network devices, convert at least a portion of Layer 3 out(L3out) content of the created local logical model and/or at least aportion of L3out content of the received software model into a commonformat, and compare content of at least some L3out overlapping fieldsfrom the common format of the created local logical model and the commonformat of the received software model. In some examples, a positiveoutcome of the comparison at least partially represents that theinternal subnet has been properly leaked outside of the fabric.

In some examples, at least some L3out overlapping fields subject to thecomparison includes leaf, port and network to thereby at least partiallyvalidate that an L3out interface has been properly deployed. In someexamples, at least some L3out overlapping fields subject to thecomparison includes network devices to thereby at least partiallyvalidate that an L3out loopback has been properly deployed. In someexamples, at least some L3out overlapping fields subject to thecomparison includes leaf, and next hop to thereby at least partiallyvalidate that L3out static routes has been properly deployed. In someexamples, at least some L3out overlapping fields subject to thecomparison includes fields to thereby at least partially validate thatendpoint groups has been properly deployed.

The systems, methods and CRM can be configured to validate each leakedinternal subnet in the longest prefix match (LPM) table of the softwaremodel has a next hop that identifies which border leaf leaked theinternal subnet, wherein a positive outcome of the validating at leastpartially represents that the internal subnet has been properly leakedoutside of the fabric.

The systems, methods and CRM can be configured to validate that an LPMtable of the border leaf has a next hop for the leaked internal subnetthat identifies the network device, wherein a positive outcome of thevalidating at least partially represents that the internal subnet hasbeen properly leaked outside of the fabric.

Also disclosed are systems, methods and CRM for performing a BD-L3outAssociation network assurance check of proper deployment of aconfiguration in a fabric. The systems, methods and CRM can beconfigured to receive, from a controller, a global logical model in afirst format, the global logical model containing instructions on howendpoints connected to a network fabric communicate within the fabric,identify bridge domain (BD) subnets in the global logical model that aredesignated as public and validate, in response to a positive result ofthe identification, that the identified BDs are associated with anL3out. In some examples, a negative outcome of the identification or apositive result of the validation at least partially represents properconfiguration a BD-Layer 3 out (L3out) relationship.

The systems, methods and CRM can also be configured to determine whetherany of the identified BDs has a different endpoint group (EPG) from itscorresponding L3out.

The systems, methods and CRM can also be configured to confirm, inresponse to a positive result of the determination, a presence of acontract between any of the identified BDs having a different endpointgroup (EPG) from its corresponding L3out.

In some examples, a positive result of the confirmation at leastpartially represents proper configuration of the BD-L3out relationship.In some examples, an error event can be generated in response to apositive outcome of the identification. In some examples, an error eventcan be generated in response to a negative result of the validation. Insome examples, an error event can be generated in response to a negativeresult of the confirmation.

Also disclosed are systems, methods and CRM for performing Learnedroutes network assurance check of proper deployment of a configurationin a fabric. The systems, methods and CRM can be configured to receive,from one or more network devices within the fabric, a software modelbeing at least a subset of instructions from the global logical model ina second format executable on the one or more network devices, thesubset of instructions being instructions from the global logical modelthat are specific to operability of the one or more network devices. Thesystems, methods and CRM can also be configured to identify from theplurality of network devices a source leaf that imported an externalsubnet from an external device, the source leaf having an L3out under avirtual routing and forwarding instance (VRF), identify from theplurality of network devices a subgroup of leafs, the subgroup of leafsincluding the source leaf and other leafs having an L3out or BD underthe VRF of the source leaf, confirm that the imported external subnet isconsistent in the software model of one or more leafs of the group ofleafs, determine, at the source leaf, the next hop of the importednetwork is the network device that requested the leak, and determine, atthe other leafs, the next hop of the imported network is at least thesource leaf. In some examples, a positive result of the determinationsand the confirming at least partially represents proper propagation ofthe imported route. In some examples, a negative result of thedeterminations represents an improper propagation of the imported route.In some examples, a negative result of the confirmation represents animproper propagation of the imported route.

The systems, methods and CRM can also be configured to confirm that theimported external subnet is consistent in the software model of allleafs of the group of leafs further comprises instructions, which whenexecuted by the at least one processor causes the at least one processorto confirm that the imported external subnet is consistent in thelongest prefix match (LPM) table of the software model in all leafs ofthe group of leafs.

The systems, methods and CRM can also be configured to receive, from acontroller, the global logical model containing instructions on howendpoints connected to a network fabric communicate within each otherthrough one or more network devices within the fabric, and confirm thatany imported routes in the global logical model are consistent with anyimported routes in the received software models of border leafs of theplurality of leafs.

The systems, methods and CRM can also be configured to confirm that anyimported routes in the global logical model are consistent with anyimported routes in received software models of border leafs of theplurality of leafs comprises instructions, which when executed by the atleast one processor, causes the at least one processor to confirm thatLPM tables of the software models of border leafs include any importedroutes found in an endpoint group (EPG) of Layer 3 out (L3out) of theglobal logical model.

The systems, methods and CRM can also be configured to confirm that anyimported routes in the global logical model is consistent with anyimported route received software models of border leafs of the pluralityof leafs comprises instructions, which when executed by the at least oneprocessor, causes the at least one processor to confirm that LPM tablesof the software models of border leafs includes any imported routesfound in an EPG of L3out of the global logical model and does notinclude other imported route unless imported from a different borderleaf or a different L3out.

DESCRIPTION

The disclosed technology addresses the need in the art for accurate andefficient discovery of problems in a large and complex network or datacenter. The present technology involves system, methods, andcomputer-readable media for network assurance in layer 1, layer 2 andlayer 3 of the networked environment. The present technology alsoinvolves system, methods, and computer-readable media for networkassurance for internal-internal (e.g., inter-fabric) forwarding andinternal-external (e.g., outside the fabric) forwarding in the networkedenvironment. The network assurance can be performed using logicalconfigurations, software configurations and/or hardware configurations.The present technology will be described in the following disclosure asfollows. The discussion begins with an introductory discussion ofnetwork assurance and fault code aggregation across application-centricdimensions. An introductory discussion of network assurance and adescription of example computing environments, as illustrated in FIGS.1A and 1B, will then follow. The discussion continues with a descriptionof systems and methods for network assurance, network modeling, andfault code aggregation across logical or application-centric dimensions,as shown in FIGS. 2A-2D, 3A-D, 4 and 5A-B. The discussion continues witha description of systems and methods for routing and forwardingassurances and checks, as shown in FIG. 3-20 . The discussion concludeswith a description of an example network device, as illustrated in FIG.21 , and an example computing device, as illustrated in FIG. 22 ,including example hardware components suitable for hosting softwareapplications and performing computing operations.

The disclosure now turns to a discussion of network assurance anddistributed fault code aggregation across logical or application-centricdimensions.

Network assurance is the guarantee or determination that the network isbehaving as intended by the network operator and has been configuredproperly (e.g., the network is doing what it is intended to do). Intentcan encompass various network operations, such as bridging, routing,security, service chaining, endpoints, compliance, QoS (Quality ofService), audits, etc. Intent can be embodied in one or more policies,settings, configurations, etc., defined for the network and individualnetwork elements (e.g., switches, routers, applications, resources,etc.). However, often times, the configurations, policies, etc., definedby a network operator are incorrect or not accurately reflected in theactual behavior of the network. For example, a network operatorspecifies a configuration A for one or more types of traffic but laterfinds out that the network is actually applying configuration B to thattraffic or otherwise processing that traffic in a manner that isinconsistent with configuration A. This can be a result of manydifferent causes, such as hardware errors, software bugs, varyingpriorities, configuration conflicts, misconfiguration of one or moresettings, improper rule rendering by devices, unexpected errors orevents, software upgrades, configuration changes, failures, etc. Asanother example, a network operator implements configuration C but oneor more other configurations result in the network behaving in a mannerthat is inconsistent with the intent reflected by the implementation ofconfiguration C. For example, such a situation can result whenconfiguration C conflicts with other configurations in the network.

The approaches herein can provide network assurance by modeling variousaspects of the network and/or performing consistency checks as well asother network assurance checks. The approaches herein can also enableidentification and visualization of hardware-level (e.g., networkswitch-level) errors along any software or application-centricdimension. Non-limiting example visualizations can include: 1)per-tenant error aggregation, 2) per-application profile erroraggregation, 3) per-endpoint group pair aggregation, and 4) per-contracterror aggregation. In this way, data center operators can quickly seehardware errors that impact particular tenants or other logicalentities, across the entire network fabric, and even drill down by otherdimensions, such as endpoint groups, to see only those relevant hardwareerrors. These visualizations speed root cause analysis, improving datacenter and application availability metrics. Given the scale of thenetwork fabric, the aggregations to create these visualizations can bedone in a distributed fashion.

In this context, a network assurance platform can run an assuranceoperator on each individual network device, such as a switch, and emitfault codes associated with the network device. A logical policyenricher can map the hardware IDs (e.g., scope, pcTag, etc.) to thelogical policy entity that is defined in the software-defined network(SDN) fabric configuration, such as the application-centricinfrastructure (ACI) fabric configuration. The mappings can yieldenriched fault codes. The enriched fault codes can be sent to anaggregation layer for aggregation. For example, multiple nodes (e.g.,HADOOP) can collect the enriched fault codes and emit them to anaggregation layer as (key, tag) pairs.

In some cases, the aggregation layer can scale horizontally by runningthe aggregator for each key as a separate reducer. Each key canrepresent a different dimension for aggregation. Non-limiting examplesof dimensions include tenant, contract, application profile, endpointgroup (EPG) pair, etc. This provides the operator of a large scalenetwork fabric with an integrated view of the health of the networkfabric for that particular dimension of aggregation. For example, thiscan provide the health of each tenant, contract, application profile,EPG pair, etc.

As previously noted, the fault code aggregation can implement logicalmodels which can represent various aspects of a network. A model caninclude a mathematical or semantic model of the network, including,without limitation the network's policies, configurations, requirements,security, routing, topology, applications, hardware, filters, contracts,access control lists, EPGs, application profiles, tenants, etc. Modelscan be implemented to provide network assurance to ensure that thenetwork is properly configured and the behavior of the network will beconsistent (or is consistent) with the intended behavior reflectedthrough specific policies, settings, definitions, etc., implemented bythe network operator. Unlike traditional network monitoring whichinvolves sending and analyzing data packets and observing networkbehavior, network assurance can be performed through modeling withoutnecessarily ingesting any packet data or monitoring traffic or networkbehavior. This can result in foresight, insight, and hindsight: problemscan be prevented before they occur, identified when they occur, andfixed immediately after they occur.

Properties of the network can be mathematically modeled todeterministically predict the behavior and condition of the network. Amathematical model can abstract the control, management, and dataplanes, and may use various techniques such as symbolic, formalverification, consistency, graph, behavioral, etc. The network can bedetermined to be healthy if the model(s) indicate proper behavior (e.g.,no inconsistencies, conflicts, errors, etc.). The network can bedetermined to be functional, but not fully healthy, if the modelingindicates proper behavior but some inconsistencies. The network can bedetermined to be non-functional and not healthy if the modelingindicates improper behavior and errors. If inconsistencies or errors aredetected by the modeling, a detailed analysis of the correspondingmodel(s) can allow one or more underlying or root problems to beidentified with great accuracy.

The models can consume numerous types of data and/or events which modela large amount of behavioral aspects of the network. Such data andevents can impact various aspects of the network, such as underlayservices, overlay service, tenant connectivity, tenant security, tenantEP mobility, tenant policy, resources, etc.

Having described various aspects of network assurance and fault codeaggregation across dimensions, the disclosure now turns to a discussionof example network environments for network assurance and fault codeaggregation.

FIG. 1A illustrates a diagram of an example Network Environment 100,such as a data center. The Network Environment 100 can include a Fabric120 which can represent the physical layer or infrastructure (e.g.,underlay) of the Network Environment 100. Fabric 120 can include Spines102 (e.g., spine routers or switches) and Leafs 104 (e.g., leaf routersor switches) which can be interconnected for routing or switchingtraffic in the Fabric 120. Spines 102 can interconnect Leafs 104 in theFabric 120, and Leafs 104 can connect the Fabric 120 to an overlay orlogical portion of the Network Environment 100, which can includeapplication services, servers, virtual machines, containers, endpoints,etc. Thus, network connectivity in the Fabric 120 can flow from Spines102 to Leafs 104, and vice versa. The interconnections between Leafs 104and Spines 102 can be redundant (e.g., multiple interconnections) toavoid a failure in routing. In some embodiments, Leafs 104 and Spines102 can be fully connected, such that any given Leaf is connected toeach of the Spines 102, and any given Spine is connected to each of theLeafs 104. Leafs 104 can be, for example, top-of-rack (“ToR”) switches,aggregation switches, gateways, ingress and/or egress switches, provideredge devices, and/or any other type of routing or switching device.

Leafs 104 can be responsible for routing and/or bridging tenant orcustomer packets and applying network policies or rules. Networkpolicies and rules can be driven by one or more Controllers 116, and/orimplemented or enforced by one or more devices, such as Leafs 104. Leafs104 can connect other elements to the Fabric 120. For example, Leafs 104can connect Servers 106, Hypervisors 108, Virtual Machines (VMs) 110,Applications 112, Network Device 114, etc., with Fabric 120. Suchelements can reside in one or more logical or virtual layers ornetworks, such as an overlay network. In some cases, Leafs 104 canencapsulate and decapsulate packets to and from such elements (e.g.,Servers 106) in order to enable communications throughout NetworkEnvironment 100 and Fabric 120. Leafs 104 can also provide any otherdevices, services, tenants, or workloads with access to Fabric 120. Insome cases, Servers 106 connected to Leafs 104 can similarly encapsulateand decapsulate packets to and from Leafs 104. For example, Servers 106can include one or more virtual switches or routers or tunnel endpointsfor tunneling packets between an overlay or logical layer hosted by, orconnected to, Servers 106 and an underlay layer represented by Fabric120 and accessed via Leafs 104.

Applications 112 can include software applications, services,containers, appliances, functions, service chains, etc. For example,Applications 112 can include a firewall, a database, a CDN server, anIDS/IPS, a deep packet inspection service, a message router, a virtualswitch, etc. An application from Applications 112 can be distributed,chained, or hosted by multiple endpoints (e.g., Servers 106, VMs 110,etc.), or may run or execute entirely from a single endpoint.

VMs 110 can be virtual machines hosted by Hypervisors 108 or virtualmachine managers running on Servers 106. VMs 110 can include workloadsrunning on a guest operating system on a respective server. Hypervisors108 can provide a layer of software, firmware, and/or hardware thatcreates, manages, and/or runs the VMs 110. Hypervisors 108 can allow VMs110 to share hardware resources on Servers 106, and the hardwareresources on Servers 106 to appear as multiple, separate hardwareplatforms. Moreover, Hypervisors 108 on Servers 106 can host one or moreVMs 110.

In some cases, VMs 110 and/or Hypervisors 108 can be migrated to otherServers 106. Servers 106 can similarly be migrated to other locations inNetwork Environment 100. For example, a server connected to a specificleaf can be changed to connect to a different or additional leaf. Suchconfiguration or deployment changes can involve modifications tosettings, configurations and policies that are applied to the resourcesbeing migrated as well as other network components.

In some cases, one or more Servers 106, Hypervisors 108, and/or VMs 110can represent or reside in a tenant or customer space. Tenant space caninclude workloads, services, applications, devices, networks, and/orresources that are associated with one or more clients or subscribers.Accordingly, traffic in Network Environment 100 can be routed based onspecific tenant policies, spaces, agreements, configurations, etc.Moreover, addressing can vary between one or more tenants. In someconfigurations, tenant spaces can be divided into logical segmentsand/or networks and separated from logical segments and/or networksassociated with other tenants. Addressing, policy, security andconfiguration information between tenants can be managed by Controllers116, Servers 106, Leafs 104, etc.

Configurations in Network Environment 100 can be implemented at alogical level, a hardware level (e.g., physical), and/or both. Forexample, configurations can be implemented at a logical and/or hardwarelevel based on endpoint or resource attributes, such as endpoint typesand/or application groups or profiles, through a software-definednetwork (SDN) framework (e.g., Application-Centric Infrastructure (ACI)or VMWARE NSX). To illustrate, one or more operators can defineconfigurations at a logical level (e.g., application or software level)through Controllers 116, which can implement or propagate suchconfigurations through Network Environment 100. In some examples,Controllers 116 can be Application Policy Infrastructure Controllers(APICs) in an ACI framework. In other examples, Controllers 116 can beone or more management components for associated with other SDNsolutions, such as NSX Managers.

Such configurations can define rules, policies, priorities, protocols,attributes, objects, etc., for routing and/or classifying traffic inNetwork Environment 100. For example, such configurations can defineattributes and objects for classifying and processing traffic based onEndpoint Groups (EPGs), Security Groups (SGs), VM types, bridge domains(BDs), virtual routing and forwarding instances (VRFs), tenants,priorities, firewall rules, etc. Other example network objects andconfigurations are further described below. Traffic policies and rulescan be enforced based on tags, attributes, or other characteristics ofthe traffic, such as protocols associated with the traffic, EPGsassociated with the traffic, SGs associated with the traffic, networkaddress information associated with the traffic, etc. Such policies andrules can be enforced by one or more elements in Network Environment100, such as Leafs 104, Servers 106, Hypervisors 108, Controllers 116,etc. As previously explained, Network Environment 100 can be configuredaccording to one or more particular software-defined network (SDN)solutions, such as CISCO ACI or VMWARE NSX. These example SDN solutionsare briefly described below.

ACI can provide an application-centric or policy-based solution throughscalable distributed enforcement. ACI supports integration of physicaland virtual environments under a declarative configuration model fornetworks, servers, services, security, requirements, etc. For example,the ACI framework implements EPGs, which can include a collection ofendpoints or applications that share common configuration requirements,such as security, QoS, services, etc. Endpoints can be virtual/logicalor physical devices, such as VMs, containers, hosts, or physical serversthat are connected to Network Environment 100. Endpoints can have one ormore attributes such as a VM name, guest OS name, a security tag,application profile, etc. Application configurations can be appliedbetween EPGs, instead of endpoints directly, in the form of contracts.Leafs 104 can classify incoming traffic into different EPGs. Theclassification can be based on, for example, a network segmentidentifier such as a VLAN ID, VXLAN Network Identifier (VNID), NVGREVirtual Subnet Identifier (VSID), MAC address, IP address, etc.

In some cases, classification in the ACI infrastructure can beimplemented by Application Virtual Switches (AVS), which can run on ahost, such as a server or switch. For example, an AVS can classifytraffic based on specified attributes, and tag packets of differentattribute EPGs with different identifiers, such as network segmentidentifiers (e.g., VLAN ID). Finally, Leafs 104 can tie packets withtheir attribute EPGs based on their identifiers and enforce policies,which can be implemented and/or managed by one or more Controllers 116.Leaf 104 can classify to which EPG the traffic from a host belongs andenforce policies accordingly.

Another example SDN solution is based on VMWARE NSX. With VMWARE NSX,hosts can run a distributed firewall (DFW) which can classify andprocess traffic. Consider a case where three types of VMs, namely,application, database and web VMs, are put into a single layer-2 networksegment. Traffic protection can be provided within the network segmentbased on the VM type. For example, HTTP traffic can be allowed among webVMs, and disallowed between a web VM and an application or database VM.To classify traffic and implement policies, VMWARE NSX can implementsecurity groups, which can be used to group the specific VMs (e.g., webVMs, application VMs, database VMs). DFW rules can be configured toimplement policies for the specific security groups. To illustrate, inthe context of the previous example, DFW rules can be configured toblock HTTP traffic between web, application, and database securitygroups.

Returning now to FIG. 1A, Network Environment 100 can deploy differenthosts via Leafs 104, Servers 106, Hypervisors 108, VMs 110, Applications112, and Controllers 116, such as VMWARE ESXi hosts, WINDOWS HYPER-Vhosts, bare metal physical hosts, etc. Network Environment 100 mayinteroperate with a variety of Hypervisors 108, Servers 106 (e.g.,physical and/or virtual servers), SDN orchestration platforms, etc.Network Environment 100 may implement a declarative model to allow itsintegration with application design and holistic network policy.

Controllers 116 can provide centralized access to fabric information,application configuration, resource configuration, application-levelconfiguration modeling for a software-defined network (SDN)infrastructure, integration with management systems or servers, etc.Controllers 116 can form a control plane that interfaces with anapplication plane via northbound APIs and a data plane via southboundAPIs.

As previously noted, Controllers 116 can define and manageapplication-level model(s) for configurations in Network Environment100. In some cases, application or device configurations can also bemanaged and/or defined by other components in the network. For example,a hypervisor or virtual appliance, such as a VM or container, can run aserver or management tool to manage software and services in NetworkEnvironment 100, including configurations and settings for virtualappliances.

As illustrated above, Network Environment 100 can include one or moredifferent types of SDN solutions, hosts, etc. For the sake of clarityand explanation purposes, various examples in the disclosure will bedescribed with reference to an ACI framework, and Controllers 116 may beinterchangeably referenced as controllers, APICs, or APIC controllers.However, it should be noted that the technologies and concepts hereinare not limited to ACI solutions and may be implemented in otherarchitectures and scenarios, including other SDN solutions as well asother types of networks which may not deploy an SDN solution.

Further, as referenced herein, the term “hosts” can refer to Servers 106(e.g., physical or logical), Hypervisors 108, VMs 110, containers (e.g.,Applications 112), etc., and can run or include any type of server orapplication solution. Non-limiting examples of “hosts” can includevirtual switches or routers, such as distributed virtual switches (DVS),application virtual switches (AVS), vector packet processing (VPP)switches; VCENTER and NSX MANAGERS; bare metal physical hosts; HYPER-Vhosts; VMs; DOCKER Containers; etc.

FIG. 1B illustrates another example of Network Environment 100. In thisexample, Network Environment 100 includes Endpoints 122 connected toLeafs 104 in Fabric 120. Endpoints 122 can be physical and/or logical orvirtual entities, such as servers, clients, VMs, hypervisors, softwarecontainers, applications, resources, network devices, workloads, etc.For example, an Endpoint 122 can be an object that represents a physicaldevice (e.g., server, client, switch, etc.), an application (e.g., webapplication, database application, etc.), a logical or virtual resource(e.g., a virtual switch, a virtual service appliance, a virtualizednetwork function (VNF), a VM, a service chain, etc.), a containerrunning a software resource (e.g., an application, an appliance, a VNF,a service chain, etc.), storage, a workload or workload engine, etc.Endpoints 122 can have an address (e.g., an identity), a location (e.g.,host, network segment, virtual routing and forwarding (VRF) instance,domain, etc.), one or more attributes (e.g., name, type, version, patchlevel, OS name, OS type, etc.), a tag (e.g., security tag), a profile,etc.

Endpoints 122 can be associated with respective Logical Groups 118.Logical Groups 118 can be logical entities containing endpoints(physical and/or logical or virtual) grouped together according to oneor more attributes, such as endpoint type (e.g., VM type, workload type,application type, etc.), one or more requirements (e.g., policyrequirements, security requirements, QoS requirements, customerrequirements, resource requirements, etc.), a resource name (e.g., VMname, application name, etc.), a profile, platform or operating system(OS) characteristics (e.g., OS type or name including guest and/or hostOS, etc.), an associated network or tenant, one or more policies, a tag,etc. For example, a logical group can be an object representing acollection of endpoints grouped together. To illustrate, Logical Group 1can contain client endpoints, Logical Group 2 can contain web serverendpoints, Logical Group 3 can contain application server endpoints,Logical Group N can contain database server endpoints, etc. In someexamples, Logical Groups 118 are EPGs in an ACI environment and/or otherlogical groups (e.g., SGs) in another SDN environment.

Traffic to and/or from Endpoints 122 can be classified, processed,managed, etc., based Logical Groups 118. For example, Logical Groups 118can be used to classify traffic to or from Endpoints 122, apply policiesto traffic to or from Endpoints 122, define relationships betweenEndpoints 122, define roles of Endpoints 122 (e.g., whether an endpointconsumes or provides a service, etc.), apply rules to traffic to or fromEndpoints 122, apply filters or access control lists (ACLs) to trafficto or from Endpoints 122, define communication paths for traffic to orfrom Endpoints 122, enforce requirements associated with Endpoints 122,implement security and other configurations associated with Endpoints122, etc.

In an ACI environment, Logical Groups 118 can be EPGs used to definecontracts in the ACI. Contracts can include rules specifying what andhow communications between EPGs take place. For example, a contract candefine what provides a service, what consumes a service, and what policyobjects are related to that consumption relationship. A contract caninclude a policy that defines the communication path and all relatedelements of a communication or relationship between endpoints or EPGs.For example, a Web EPG can provide a service that a Client EPG consumes,and that consumption can be subject to a filter (ACL) and a servicegraph that includes one or more services, such as firewall inspectionservices and server load balancing.

FIG. 2A illustrates a diagram of an example Management Information Model200 for an SDN network, such as Network Environment 100. The followingdiscussion of Management Information Model 200 references various termswhich shall also be used throughout the disclosure. Accordingly, forclarity, the disclosure shall first provide below a list of terminology,which will be followed by a more detailed discussion of ManagementInformation Model 200.

As used herein, an “Alias” can refer to a changeable name for a givenobject. Thus, even if the name of an object, once created, cannot bechanged, the Alias can be a field that can be changed.

As used herein, the term “Aliasing” can refer to a rule (e.g.,contracts, policies, configurations, etc.) that overlaps one or moreother rules. For example, Contract 1 defined in a logical model of anetwork can be said to be aliasing Contract 2 defined in the logicalmodel of the network if Contract 1 overlaps Contract 1. In this example,by aliasing Contract 2, Contract 1 may render Contract 2 redundant orinoperable. For example, if Contract 1 has a higher priority thanContract 2, such aliasing can render Contract 2 redundant based onContract 1's overlapping and higher priority characteristics.

As used herein, the term “APIC” can refer to one or more controllers(e.g., Controllers 116) in an ACI framework. The APIC can provide aunified point of automation and management, policy programming,application deployment, health monitoring for an ACI multitenant fabric.The APIC can be implemented as a single controller, a distributedcontroller, or a replicated, synchronized, and/or clustered controller.

As used herein, the term “BDD” can refer to a binary decision tree Abinary decision tree can be a data structure representing functions,such as Boolean functions.

As used herein, the term “BD” can refer to a bridge domain. A bridgedomain can be a set of logical ports that share the same flooding orbroadcast characteristics. Like a virtual LAN (VLAN), bridge domains canspan multiple devices. A bridge domain can be a L2 (Layer 2) construct.

As used herein, a “Consumer” can refer to an endpoint, resource, and/orEPG that consumes a service.

As used herein, a “Context” can refer to an L3 (Layer 3) address domainthat allows multiple instances of a routing table to exist and worksimultaneously. This increases functionality by allowing network pathsto be segmented without using multiple devices. Non-limiting examples ofa context or L3 address domain can include a Virtual Routing andForwarding (VRF) instance, a private network, and so forth.

As used herein, the term “Contract” can refer to rules or configurationsthat specify what and how communications in a network are conducted(e.g., allowed, denied, filtered, processed, etc.). In an ACI network,contracts can specify how communications between endpoints and/or EPGstake place. In some examples, a contract can provide rules andconfigurations akin to an Access Control List (ACL).

As used herein, the term “Distinguished Name” (DN) can refer to a uniquename that describes an object, such as an MO, and locates its place inManagement Information Model 200. In some cases, the DN can be (orequate to) a Fully Qualified Domain Name (FQDN).

As used herein, the term “Endpoint Group” (EPG) can refer to a logicalentity or object associated with a collection or group of endpoints aspreviously described with reference to FIG. 1B.

As used herein, the term “Filter” can refer to a parameter orconfiguration for allowing communications. For example, in a whitelistmodel where all communications are blocked by default, a communicationmust be given explicit permission to prevent such communication frombeing blocked. A filter can define permission(s) for one or morecommunications or packets. A filter can thus function similar to an ACLor Firewall rule. In some examples, a filter can be implemented in apacket (e.g., TCP/IP) header field, such as L3 protocol type, L4 (Layer4) ports, and so on, which is used to allow inbound or outboundcommunications between endpoints or EPGs, for example.

As used herein, the term “L2 Out” can refer to a bridged connection. Abridged connection can connect two or more segments of the same networkso that they can communicate. In an ACI framework, an L2 out can be abridged (Layer 2) connection between an ACI fabric (e.g., Fabric 120)and an outside Layer 2 network, such as a switch.

As used herein, the term “L3 Out” can refer to a routed connection. Arouted Layer 3 connection uses a set of protocols that determine thepath that data follows in order to travel across networks from itssource to its destination. Routed connections can perform forwarding(e.g., IP forwarding) according to a protocol selected, such as BGP(border gateway protocol), OSPF (Open Shortest Path First), EIGRP(Enhanced Interior Gateway Routing Protocol), etc.

As used herein, the term “Managed Object” (MO) can refer to an abstractrepresentation of objects that are managed in a network (e.g., NetworkEnvironment 100). The objects can be concrete objects (e.g., a switch,server, adapter, etc.), or logical objects (e.g., an applicationprofile, an EPG, a fault, etc.). The MOs can be network resources orelements that are managed in the network. For example, in an ACIenvironment, an MO can include an abstraction of an ACI fabric (e.g.,Fabric 120) resource.

As used herein, the term “Management Information Tree” (MIT) can referto a hierarchical management information tree containing the MOs of asystem. For example, in ACI, the MIT contains the MOs of the ACI fabric(e.g., Fabric 120). The MIT can also be referred to as a ManagementInformation Model (MIM), such as Management Information Model 200.

As used herein, the term “Policy” can refer to one or morespecifications for controlling some aspect of system or networkbehavior. For example, a policy can include a named entity that containsspecifications for controlling some aspect of system behavior. Toillustrate, a Layer 3 Outside Network Policy can contain the BGPprotocol to enable BGP routing functions when connecting Fabric 120 toan outside Layer 3 network.

As used herein, the term “Profile” can refer to the configurationdetails associated with a policy. For example, a profile can include anamed entity that contains the configuration details for implementingone or more instances of a policy. To illustrate, a switch node profilefor a routing policy can contain the switch-specific configurationdetails to implement the BGP routing protocol.

As used herein, the term “Provider” refers to an object or entityproviding a service. For example, a provider can be an EPG that providesa service.

As used herein, the term “Subject” refers to one or more parameters in acontract for defining communications. For example, in ACI, subjects in acontract can specify what information can be communicated and how.Subjects can function similar to ACLs.

As used herein, the term “Tenant” refers to a unit of isolation in anetwork. For example, a tenant can be a secure and exclusive virtualcomputing environment. In ACI, a tenant can be a unit of isolation froma policy perspective, but does not necessarily represent a privatenetwork. Indeed, ACI tenants can contain multiple private networks(e.g., VRFs). Tenants can represent a customer in a service providersetting, an organization or domain in an enterprise setting, or just agrouping of policies.

As used herein, the term “VRF” refers to a virtual routing andforwarding instance. The VRF can define a Layer 3 address domain thatallows multiple instances of a routing table to exist and worksimultaneously. This increases functionality by allowing network pathsto be segmented without using multiple devices. Also known as a contextor private network.

Having described various terms used herein, the disclosure now returnsto a discussion of Management Information Model (MIM) 200 in FIG. 2A. Aspreviously noted, MIM 200 can be a hierarchical management informationtree or MIT. Moreover, MIM 200 can be managed and processed byControllers 116, such as APICs in an ACI. Controllers 116 can enable thecontrol of managed resources by presenting their manageablecharacteristics as object properties that can be inherited according tothe location of the object within the hierarchical structure of themodel.

The hierarchical structure of MIM 200 starts with Policy Universe 202 atthe top (Root) and contains parent and child nodes 116, 204, 206, 208,210, 212. Nodes 116, 202, 204, 206, 208, 210, 212 in the tree representthe managed objects (MOs) or groups of objects. Each object in thefabric (e.g., Fabric 120) has a unique distinguished name (DN) thatdescribes the object and locates its place in the tree. The Nodes 116,202, 204, 206, 208, 210, 212 can include the various MOs, as describedbelow, which contain policies that govern the operation of the system.

Controllers 116

Controllers 116 (e.g., APIC controllers) can provide management, policyprogramming, application deployment, and health monitoring for Fabric120.

Node 204

Node 204 includes a tenant container for policies that enable anoperator to exercise domain-based access control. Non-limiting examplesof tenants can include:

User tenants defined by the operator according to the needs of users.They contain policies that govern the operation of resources such asapplications, databases, web servers, network-attached storage, virtualmachines, and so on.

The common tenant is provided by the system but can be configured by theoperator. It contains policies that govern the operation of resourcesaccessible to all tenants, such as firewalls, load balancers, Layer 4 toLayer 7 services, intrusion detection appliances, and so on.

The infrastructure tenant is provided by the system but can beconfigured by the operator. It contains policies that govern theoperation of infrastructure resources such as the fabric overlay (e.g.,VXLAN). It also enables a fabric provider to selectively deployresources to one or more user tenants. Infrastructure tenant polices canbe configurable by the operator.

The management tenant is provided by the system but can be configured bythe operator. It contains policies that govern the operation of fabricmanagement functions used for in-band and out-of-band configuration offabric nodes. The management tenant contains a private out-of-boundaddress space for the Controller/Fabric internal communications that isoutside the fabric data path that provides access through the managementport of the switches. The management tenant enables discovery andautomation of communications with virtual machine controllers.

Node 206

Node 206 can contain access policies that govern the operation of switchaccess ports that provide connectivity to resources such as storage,compute, Layer 2 and Layer 3 (bridged and routed) connectivity, virtualmachine hypervisors, Layer 4 to Layer 7 devices, and so on. If a tenantrequires interface configurations other than those provided in thedefault link, Cisco Discovery Protocol (CDP), Link Layer DiscoveryProtocol (LLDP), Link Aggregation Control Protocol (LACP), or SpanningTree Protocol (STP), an operator can configure access policies to enablesuch configurations on the access ports of Leafs 104.

Node 206 can contain fabric policies that govern the operation of theswitch fabric ports, including such functions as Network Time Protocol(NTP) server synchronization, Intermediate System-to-Intermediate SystemProtocol (IS-IS), Border Gateway Protocol (BGP) route reflectors, DomainName System (DNS) and so on. The fabric MO contains objects such aspower supplies, fans, chassis, and so on.

Node 208

Node 208 can contain VM domains that group VM controllers with similarnetworking policy requirements. VM controllers can share virtual space(e.g., VLAN or VXLAN space) and application EPGs. Controllers 116communicate with the VM controller to publish network configurationssuch as port groups that are then applied to the virtual workloads.

Node 210

Node 210 can contain Layer 4 to Layer 7 service integration life cycleautomation framework that enables the system to dynamically respond whena service comes online or goes offline. Policies can provide servicedevice package and inventory management functions.

Node 212

Node 212 can contain access, authentication, and accounting (AAA)policies that govern user privileges, roles, and security domains ofFabric 120.

The hierarchical policy model can fit well with an API, such as a RESTAPI interface. When invoked, the API can read from or write to objectsin the MIT. URLs can map directly into distinguished names that identifyobjects in the MIT. Data in the MIT can be described as a self-containedstructured tree text document encoded in XML or JSON, for example.

FIG. 2B illustrates an example object model 220 for a tenant portion ofMIM 200. As previously noted, a tenant is a logical container forapplication policies that enable an operator to exercise domain-basedaccess control. A tenant thus represents a unit of isolation from apolicy perspective, but it does not necessarily represent a privatenetwork. Tenants can represent a customer in a service provider setting,an organization or domain in an enterprise setting, or just a convenientgrouping of policies. Moreover, tenants can be isolated from one anotheror can share resources.

Tenant portion 204A of MIM 200 can include various entities, and theentities in Tenant Portion 204A can inherit policies from parententities. Non-limiting examples of entities in Tenant Portion 204A caninclude Filters 240, Contracts 236, Outside Networks 222, Bridge Domains230, VRF Instances 234, and Application Profiles 224.

Bridge Domains 230 can include Subnets 232. Contracts 236 can includeSubjects 238. Application Profiles 224 can contain one or more EPGs 226.Some applications can contain multiple components. For example, ane-commerce application could require a web server, a database server,data located in a storage area network, and access to outside resourcesthat enable financial transactions. Application Profile 224 contains asmany (or as few) EPGs as necessary that are logically related toproviding the capabilities of an application.

EPG 226 can be organized in various ways, such as based on theapplication they provide, the function they provide (such asinfrastructure), where they are in the structure of the data center(such as DMZ), or whatever organizing principle that a fabric or tenantoperator chooses to use.

EPGs in the fabric can contain various types of EPGs, such asapplication EPGs, Layer 2 external outside network instance EPGs, Layer3 external outside network instance EPGs, management EPGs forout-of-band or in-band access, etc. EPGs 226 can also contain Attributes228, such as encapsulation-based EPGs, IP-based EPGs, or MAC-based EPGs.

As previously mentioned, EPGs can contain endpoints (e.g., EPs 122) thathave common characteristics or attributes, such as common policyrequirements (e.g., security, virtual machine mobility (VMM), QoS, orLayer 4 to Layer 7 services). Rather than configure and manage endpointsindividually, they can be placed in an EPG and managed as a group.

Policies apply to EPGs, including the endpoints they contain. An EPG canbe statically configured by an operator in Controllers 116, ordynamically configured by an automated system such as VCENTER orOPENSTACK.

To activate tenant policies in Tenant Portion 204A, fabric accesspolicies should be configured and associated with tenant policies.Access policies enable an operator to configure other networkconfigurations, such as port channels and virtual port channels,protocols such as LLDP, CDP, or LACP, and features such as monitoring ordiagnostics.

FIG. 2C illustrates an example Association 260 of tenant entities andaccess entities in MIM 200. Policy Universe 202 contains Tenant Portion204A and Access Portion 206A. Thus, Tenant Portion 204A and AccessPortion 206A are associated through Policy Universe 202.

Access Portion 206A can contain fabric and infrastructure accesspolicies. Typically, in a policy model, EPGs are coupled with VLANs. Fortraffic to flow, an EPG is deployed on a leaf port with a VLAN in aphysical, VMM, L2 out, L3 out, or Fiber Channel domain, for example.

Access Portion 206A thus contains Domain Profile 236 which can define aphysical, VMM, L2 out, L3 out, or Fiber Channel domain, for example, tobe associated to the EPGs. Domain Profile 236 contains VLAN InstanceProfile 238 (e.g., VLAN pool) and Attachable Access Entity Profile (AEP)240, which are associated directly with application EPGs. The AEP 240deploys the associated application EPGs to the ports to which it isattached, and automates the task of assigning VLANs. While a large datacenter can have thousands of active VMs provisioned on hundreds ofVLANs, Fabric 120 can automatically assign VLAN IDs from VLAN pools.This saves time compared with trunking down VLANs in a traditional datacenter.

FIG. 2D illustrates a schematic diagram of example models for a network,such as Network Environment 100. The models can be generated based onspecific configurations and/or network state parameters associated withvarious objects, policies, properties, and elements defined in MIM 200.The models can be implemented for network analysis and assurance, andmay provide a depiction of the network at various stages ofimplementation and levels of the network.

As illustrated, the models can include L_Model 270A (Logical Model),LR_Model 270B (Logical Rendered Model or Logical Runtime Model),Li_Model 272 (Logical Model for i), Ci_Model 274 (Concrete model for i),and/or Hi_Model 276 (Hardware model or TCAM Model for i).

L_Model 270A is the logical representation of various elements in MIM200 as configured in a network (e.g., Network Environment 100), such asobjects, object properties, object relationships, and other elements inMIM 200 as configured in a network. L_Model 270A can be generated byControllers 116 based on configurations entered in Controllers 116 forthe network, and thus represents the logical configuration of thenetwork at Controllers 116. This is the declaration of the “end-state”expression that is desired when the elements of the network entities(e.g., applications, tenants, etc.) are connected and Fabric 120 isprovisioned by Controllers 116. Because L_Model 270A represents theconfigurations entered in Controllers 116, including the objects andrelationships in MIM 200, it can also reflect the “intent” of theoperator: how the operator wants the network and network elements tobehave.

L_Model 270A can be a fabric or network-wide logical model. For example,L_Model 270A can account configurations and objects from each ofControllers 116. As previously explained, Network Environment 100 caninclude multiple Controllers 116. In some cases, two or more Controllers116 may include different configurations or logical models for thenetwork. In such cases, L_Model 270A can obtain any of theconfigurations or logical models from Controllers 116 and generate afabric or network wide logical model based on the configurations andlogical models from all Controllers 116. L_Model 270A can thusincorporate configurations or logical models between Controllers 116 toprovide a comprehensive logical model. L_Model 270A can also address oraccount for any dependencies, redundancies, conflicts, etc., that mayresult from the configurations or logical models at the differentControllers 116.

L_Model 270A is thus the first/highest level model. L_Model 270A iscreated by the Controllers 116 based on input from the operator. Thecontent within L_Model 270A is thus the specific instructions on howvarious Leafs 104 and the Spines 102 within the Fabric 120 are tocommunicate with Endpoints 122. In some examples, any particular L_Model270A can be “global” in that it can be generated at a central point toinclude instructions for all of the Leafs 104 and the Spines 102, oronly some of them (in theory it could be as small as one Leaf 104 orSpine 120, although this would be unusual). The format is based onwhatever the GUI interface and/or the Rest API allows the user to inputthe content. Multiple L_Model 270A configurations may be present.

The format of L_Model 270A is in the program format of the user togenerate the instructions, and not useable as is by Controller 116.Controller 116 converts L_Model 270A as a first level into LR_Model 270Bas a second level, for which LR_Model 270B is a logical representationthat contains the content of L-Model 270A in a format that is readableon Controller 116 and transmittable to Leafs 104 and Spines 102. As thisis a format change, the information content of L_Model 270A and LR_Model270B can be the same, although the application is not so limited and thecontent can be omitted or added to create LR_Model 270B.

LR_Model 270B is thus the abstract model expression that Controllers 116(e.g., APICs in ACI) resolve from L_Model 270A. LR_Model 270B canprovide the configuration components that would be delivered to thephysical infrastructure (e.g., Fabric 120) to execute one or morepolicies. For example, LR_Model 270B can be delivered to Leafs 104 inFabric 120 to configure Leafs 104 for communication with attachedEndpoints 122. LR_Model 270B can also incorporate state information tocapture a runtime state of the network (e.g., Fabric 120).

In some cases, LR_Model 270B can provide a representation of L_Model270A that is normalized according to a specific format or expressionthat can be propagated to, and/or understood by, the physicalinfrastructure of Fabric 120 (e.g., Leafs 104, Spines 102, etc.). Forexample, LR_Model 270B can associate the elements in L_Model 270A withspecific identifiers or tags that can be interpreted and/or compiled bythe switches in Fabric 120, such as hardware plane identifiers used asclassifiers.

Controller 116 transmits L_Model 270A and/or LR_Model 270B (referred toherein individually and collectively as “L_Model 270A/B”) to relevantLeafs 104 and Spines 102 in the Fabric 120. Relevance may be defined byController 116 to be all of the Leafs 104 and Spines 102, or some subsetthereof. By way of non-limiting example, Fabric 120 of FIG. 1B includesLeaf 1, Leaf 2 and Leaf N. An L_Model 270A/B at Controller 116 may onlyinclude content for Leaf 1 and Leaf N, but not Leaf 2. Controller 116could thus transmit the L_Model 270A/B to all of Leafs 1, 2 and N, oronly to relevant Leafs 1 and N.

Controller 116 and/or or each particular Leaf 104 and Spine 102 thatreceives L_Model 270A/B then extracts/isolates content from the receivedmodel to form a local subset of content that is specific to thatparticular Leaf or Spine. This extracted/isolated content defines thethird level Li_Model 272, which is a switch-level or switch-specificlogical model; Li_Model 272 is thus “local” in that is specific to thatswitch (although it may include content on the switch's relationshipwith other switches). Li_Model 272 is thus a switch-level orswitch-specific model obtained from L_Model 270A and/or LR_Model 270B.Li_Model 272 can project L_Model 270A and/or LR_Model 270B on a specificswitch or device i, and thus can convey how L_Model 270A and/or LR_Model270B should appear or be implemented at the specific switch or device i.

For example, Li_Model 272 can project L_Model 270A and/or LR_Model 270Bpertaining to a specific switch i to capture a switch-levelrepresentation of L_Model 270A and/or LR_Model 270B at switch i. Toillustrate, Li_Model 272 L1 can represent L_Model 270A and/or LR_Model270B projected to, or implemented at, Leaf 1 (104). Thus, Li_Model 272can be generated from L_Model 270A and/or LR_Model 270B for individualdevices (e.g., Leafs 104, Spines 102, etc.) on Fabric 120. By way ofnon-limiting example, if the L_Model 270A/B includes instructions forLeaf 1 and Leaf N, then Leaf 1 will extract/isolate the content relevantto the operation of Leaf 1 to define local L1_Model 272, and Leaf N willextract/isolate the content relevant to the operation of Leaf N todefine local LN_Model 272. Since the L_Model 270A/B did not includeinstructions for Leaf 2, then even if Leaf 2 received the L_Model 270A/Bit would not create its own local L2_Model, although the application isnot so limited. To the extent that the local Li_Model 272 is a subset ofL_Model 270A/B, then this represents a change of content. To the extentthat local Li_Model 272 is an extraction/isolation of the content fromLR_Model 270B, there is generally no change in format, although theapplication is not so limited.

In some cases, Li_Model 272 can be represented using JSON (JavaScriptObject Notation). For example, Li_Model 272 can include JSON objects,such as Rules, Filters, Entries, and Scopes.

Each Li_Model 272 may not be in a format executable by the particularLeaf 104 or Spine 102 on which it was created (e.g., it may not beexecutable on the local operating system of that Leaf or Spine). EachLeaf 104 and Spine 102 can thus locally create the fourth level byconverting the format of the Li_Model 272 into a format that theoperating system of that particular Leaf 104 or Spine 102 can execute.This results in Ci_Model 274, referred to herein as a software model ora concrete model. Thus, by way of non-limiting example, where Leaf Ncreated a local LN_Model 272 from L_Model 270A/B, Leaf N will in turncreate a local CN_Model 274 from LN_Model 272.

Ci_Model 274 is thus the actual in-state configuration at the individualfabric member i (e.g., switch i). In other words, Ci_Model 274 is aswitch-level or switch-specific model that is based on Li_Model 272. Forexample, Controllers 116 can deliver Li_Model 272 to Leaf 1 (104). Leaf1 (104) can take Li_Model 272, which can be specific to Leaf 1 (104),and render the policies in Li_Model 272 into a concrete model, Ci_Model274, that runs on Leaf 1 (104). Leaf 1 (104) can render Li_Model 272 viathe OS on Leaf 1 (104), for example. Thus, Ci_Model 274 can be analogousto compiled software, as it is the form of Li_Model 272 that the switchOS at Leaf 1 (104) can execute.

The creation of local Li_Model 274 from Ci_Model 274 represents at leasta change in format. The content of the Li_Model 272 may be the sameCi_Model 274 such that there is no content change, although theapplication is not so limited and the content may only overlap. By wayof example, not all of the content from Li_Model 272 may be included inthe corresponding Ci_Model 274. Similarly, the particular Leaf 104 orSpine 102 can add content to Ci_Model 274 that was not present inLi_Model Model 272.

In some cases, Li_Model 272 and Ci_Model 274 can have a same or similarformat. For example, Li_Model 272 and Ci_Model 274 can be based on JSONobjects. Having the same or similar format can facilitate objects inLi_Model 272 and Ci_Model 274 to be compared for equivalence orcongruence. Such equivalence or congruence checks can be used fornetwork analysis and assurance, as further described herein.

Each Leaf 104 and Spine 102 that creates a Ci_Model 274 as a fourthlevel will in turn create a fifth level as a local hardware modelHi_Model 276. Hi_Model 276 represents the actual hardware configurationsfrom Ci_Model 274 that are extracted and stored in the local memory. Theconversion of Ci_Model 274 into Hi_Model 276 represents both a formatand content change. Similarly, the particular Leaf 104 or Spine 102 canadd content to Hi_Model 276 that was not present in Ci_Model Model 274.

Hi_Model 276 is thus also a switch-level or switch-specific model forswitch i, but is based on Ci_Model 274 for switch i. Hi_Model 276 is theactual configuration (e.g., rules) stored or rendered on the hardware ormemory (e.g., TCAM memory) at the individual fabric member i (e.g.,switch i). For example, Hi_Model 276 can represent the configurations(e.g., rules) which Leaf 1 (104) stores or renders on the hardware(e.g., TCAM memory) of Leaf 1 (104) based on Ci_Model 274 at Leaf 1(104). The switch OS at Leaf 1 (104) can render or execute Ci_Model 274,and Leaf 1 (104) can store or render the configurations from Ci_Model274 in storage, such as the memory or TCAM at Leaf 1 (104). Theconfigurations from Hi_Model 276 stored or rendered by Leaf 1 (104)represent the configurations that will be implemented by Leaf 1 (104)when processing traffic.

While Models 272, 274, 276 are shown as device-specific models, similarmodels can be generated or aggregated for a collection of fabric members(e.g., Leafs 104 and/or Spines 102) in Fabric 120. When combined,device-specific models, such as Model 272, Model 274, and/or Model 276,can provide a representation of Fabric 120 that extends beyond aparticular device. For example, in some cases, Li_Model 272, Ci_Model274, and/or Hi_Model 276 associated with some or all individual fabricmembers (e.g., Leafs 104 and Spines 102) can be combined or aggregatedto generate one or more aggregated models based on the individual fabricmembers.

As referenced herein, the terms H Model, T Model, and TCAM Model can beused interchangeably to refer to a hardware model, such as Hi_Model 276.For example, Ti Model, Hi Model and TCAMi Model may be usedinterchangeably to refer to Hi_Model 276.

Models 270A, 270B, 272, 274, 276 can provide representations of variousaspects of the network or various configuration stages for MIM 200. Forexample, one or more of Models 270A, 270B, 272, 274, 276 can be used togenerate Underlay Model 278 representing one or more aspects of Fabric120 (e.g., underlay topology, routing, etc.), Overlay Model 280representing one or more aspects of the overlay or logical segment(s) ofNetwork Environment 100 (e.g., COOP, MPBGP, tenants, VRFs, VLANs,VXLANs, virtual applications, VMs, hypervisors, virtual switching,etc.), Tenant Model 282 representing one or more aspects of Tenantportion 204A in MIM 200 (e.g., security, forwarding, service chaining,QoS, VRFs, BDs, Contracts, Filters, EPGs, subnets, etc.), ResourcesModel 284 representing one or more resources in Network Environment 100(e.g., storage, computing, VMs, port channels, physical elements, etc.),etc.

In general, L_Model 270A can be the high-level expression of what existsin the LR_Model 270B, which should be present on the concrete devices asCi_Model 274 and Hi_Model 276 expression. If there is any gap betweenthe models, there may be inconsistent configurations or problems.

FIG. 3A illustrates a diagram of an example Assurance Appliance 300 fornetwork assurance. In this example, Assurance Appliance 300 can includek VMs 110 operating in cluster mode. VMs are used in this example forexplanation purposes. However, it should be understood that otherconfigurations are also contemplated herein, such as use of containers,bare metal devices, Endpoints 122, or any other physical or logicalsystems. Moreover, while FIG. 3A illustrates a cluster modeconfiguration, other configurations are also contemplated herein, suchas a single mode configuration (e.g., single VM, container, or server)or a service chain for example.

Assurance Appliance 300 can run on one or more Servers 106, VMs 110,Hypervisors 108, EPs 122, Leafs 104, Controllers 116, or any othersystem or resource. For example, Assurance Appliance 300 can be alogical service or application running on one or more VMs 110 in NetworkEnvironment 100.

The Assurance Appliance 300 can include Data Framework 308, which can bebased on, for example, APACHE APEX and HADOOP. In some cases, assurancechecks can be written as individual operators that reside in DataFramework 308. This enables a natively horizontal scale-out architecturethat can scale to arbitrary number of switches in Fabric 120 (e.g., ACIfabric).

Assurance Appliance 300 can poll Fabric 120 at a configurableperiodicity (e.g., an epoch). The analysis workflow can be setup as aDAG (Directed Acyclic Graph) of Operators 310, where data flows from oneoperator to another and eventually results are generated and persistedto Database 302 for each interval (e.g., each epoch).

The north-tier implements API Server (e.g., APACHE Tomcat and Springframework) 304 and Web Server 306. A graphical user interface (GUI)interacts via the APIs exposed to the customer. These APIs can also beused by the customer to collect data from Assurance Appliance 300 forfurther integration into other tools.

Operators 310 in Data Framework 308 (e.g., APEX/Hadoop) can togethersupport assurance operations. Below are non-limiting examples ofassurance operations that can be performed by Assurance Appliance 300via Operators 310.

Security Policy Adherence

Assurance Appliance 300 can check to make sure the configurations orspecification from L_Model 270A, which may reflect the user's intent forthe network, including for example the security policies andcustomer-configured contracts, are correctly implemented and/or renderedin Li_Model 272, Ci_Model 274, and Hi_Model 276, and thus properlyimplemented and rendered by the fabric members (e.g., Leafs 104), andreport any errors, contract violations, or irregularities found.

Static Policy Analysis

Assurance Appliance 300 can check for issues in the specification of theuser's intent or intents (e.g., identify contradictory or conflictingpolicies in L_Model 270A).

TCAM Utilization

TCAM is a scarce resource in the fabric (e.g., Fabric 120). However,Assurance Appliance 300 can analyze the TCAM utilization by the networkdata (e.g., Longest Prefix Match (LPM) tables, routing tables, VLANtables, BGP updates, etc.), Contracts, Logical Groups 118 (e.g., EPGs),Tenants, Spines 102, Leafs 104, and other dimensions in NetworkEnvironment 100 and/or objects in MIM 200, to provide a network operatoror user visibility into the utilization of this scarce resource. Thiscan greatly help for planning and other optimization purposes.

Endpoint Checks

Assurance Appliance 300 can validate that the fabric (e.g. fabric 120)has no inconsistencies in the Endpoint information registered (e.g., twoleafs announcing the same endpoint, duplicate subnets, etc.), amongother such checks.

Tenant Routing Checks

Assurance Appliance 300 can validate that BDs, VRFs, subnets (bothinternal and external), VLANs, contracts, filters, applications, EPGs,etc., are correctly programmed.

Infrastructure Routing

Assurance Appliance 300 can validate that infrastructure routing (e.g.,IS-IS protocol) has no convergence issues leading to black holes, loops,flaps, and other problems.

MP-BGP Route Reflection Checks

The network fabric (e.g., Fabric 120) can interface with other externalnetworks and provide connectivity to them via one or more protocols,such as Border Gateway Protocol (BGP), Open Shortest Path First (OSPF),etc. The learned routes are advertised within the network fabric via,for example, MP-BGP. These checks can ensure that a route reflectionservice via, for example, MP-BGP (e.g., from Border Leaf) does not havehealth issues.

Logical Lint and Real-time Change Analysis

Assurance Appliance 300 can validate rules in the specification of thenetwork (e.g., L_Model 270A) are complete and do not haveinconsistencies or other problems. MOs in the MIM 200 can be checked byAssurance Appliance 300 through syntactic and semantic checks performedon L_Model 270A and/or the associated configurations of the MOs in MIM200. Assurance Appliance 300 can also verify that unnecessary, stale,unused or redundant configurations, such as contracts, are removed.

FIG. 3B illustrates an architectural diagram of an example system 350for network assurance, such as Assurance Appliance 300. In some cases,system 350 can correspond to the DAG of Operators 310 previouslydiscussed with respect to FIG. 3A

In this example, Topology Explorer 312 communicates with Controllers 116(e.g., APIC controllers) in order to discover or otherwise construct acomprehensive topological view of Fabric 120 (e.g., Spines 102, Leafs104, Controllers 116, Endpoints 122, and any other components as well astheir interconnections). While various architectural components arerepresented in a singular, boxed fashion, it is understood that a givenarchitectural component, such as Topology Explorer 312, can correspondto one or more individual Operators 310 and may include one or morenodes or endpoints, such as one or more servers, VMs, containers,applications, service functions (e.g., functions in a service chain orvirtualized network function), etc.

Topology Explorer 312 is configured to discover nodes in Fabric 120,such as Controllers 116, Leafs 104, Spines 102, etc. Topology Explorer312 can additionally detect a majority election performed amongstControllers 116, and determine whether a quorum exists amongstControllers 116. If no quorum or majority exists, Topology Explorer 312can trigger an event and alert a user that a configuration or othererror exists amongst Controllers 116 that is preventing a quorum ormajority from being reached. Topology Explorer 312 can detect Leafs 104and Spines 102 that are part of Fabric 120 and publish theircorresponding out-of-band management network addresses (e.g., IPaddresses) to downstream services. This can be part of the topologicalview that is published to the downstream services at the conclusion ofTopology Explorer's 312 discovery epoch (e.g., 5 minutes, or some otherspecified interval).

In some examples, Topology Explorer 312 can receive as input a list ofControllers 116 (e.g., APIC controllers) that are associated with thenetwork/fabric (e.g., Fabric 120). Topology Explorer 312 can alsoreceive corresponding credentials to login to each controller. TopologyExplorer 312 can retrieve information from each controller using, forexample, REST calls. Topology Explorer 312 can obtain from eachcontroller a list of nodes (e.g., Leafs 104 and Spines 102), and theirassociated properties, that the controller is aware of. TopologyExplorer 312 can obtain node information from Controllers 116 including,without limitation, an IP address, a node identifier, a node name, anode domain, a node URI, a node_dm, a node role, a node version, etc.

Topology Explorer 312 can also determine if Controllers 116 are inquorum, or are sufficiently communicatively coupled amongst themselves.For example, if there are n controllers, a quorum condition might be metwhen (n/2+1) controllers are aware of each other and/or arecommunicatively coupled. Topology Explorer 312 can make thedetermination of a quorum (or identify any failed nodes or controllers)by parsing the data returned from the controllers, and identifyingcommunicative couplings between their constituent nodes. TopologyExplorer 312 can identify the type of each node in the network, e.g.spine, leaf, APIC, etc., and include this information in the topologyinformation generated (e.g., topology map or model).

If no quorum is present, Topology Explorer 312 can trigger an event andalert a user that reconfiguration or suitable attention is required. Ifa quorum is present, Topology Explorer 312 can compile the networktopology information into a JSON object and pass it downstream to otheroperators or services, such as Unified Collector 314.

Unified Collector 314 can receive the topological view or model fromTopology Explorer 312 and use the topology information to collectinformation for network assurance from Fabric 120. Unified Collector 314can poll nodes (e.g., Controllers 116, Leafs 104, Spines 102, etc.) inFabric 120 to collect information from the nodes.

Unified Collector 314 can include one or more collectors (e.g.,collector devices, operators, applications, VMs, etc.) configured tocollect information from Topology Explorer 312 and/or nodes in Fabric120. For example, Unified Collector 314 can include a cluster ofcollectors, and each of the collectors can be assigned to a subset ofnodes within the topological model and/or Fabric 120 in order to collectinformation from their assigned subset of nodes. For performance,Unified Collector 314 can run in a parallel, multi-threaded fashion.

Unified Collector 314 can perform load balancing across individualcollectors in order to streamline the efficiency of the overallcollection process. Load balancing can be optimized by managing thedistribution of subsets of nodes to collectors, for example by randomlyhashing nodes to collectors.

In some cases, Assurance Appliance 300 can run multiple instances ofUnified Collector 314. This can also allow Assurance Appliance 300 todistribute the task of collecting data for each node in the topology(e.g., Fabric 120 including Spines 102, Leafs 104, Controllers 116,etc.) via sharding and/or load balancing, and map collection tasksand/or nodes to a particular instance of Unified Collector 314 with datacollection across nodes being performed in parallel by various instancesof Unified Collector 314. Within a given node, commands and datacollection can be executed serially. Assurance Appliance 300 can controlthe number of threads used by each instance of Unified Collector 314 topoll data from Fabric 120.

Unified Collector 314 can collect models (e.g., L_Model 270A and/orLR_Model 270B) from Controllers 116, switch software configurations andmodels (e.g., Ci_Model 274) from nodes (e.g., Leafs 104 and/or Spines102) in Fabric 120, hardware configurations and models (e.g., Hi_Model276) from nodes (e.g., Leafs 104 and/or Spines 102) in Fabric 120, etc.Unified Collector 314 can collect Ci_Model 274 and Hi_Model 276 fromindividual nodes or fabric members, such as Leafs 104 and Spines 102,and L_Model 270A and/or LR_Model 270B from one or more controllers(e.g., Controllers 116) in Network Environment 100.

Unified Collector 314 can poll the devices that Topology Explorer 312discovers in order to collect data from Fabric 120 (e.g., from theconstituent members of the fabric). Unified Collector 314 can collectthe data using interfaces exposed by Controllers 116 and/or switchsoftware (e.g., switch OS), including, for example, a RepresentationState Transfer (REST) Interface and a Secure Shell (SSH) Interface.

In some cases, Unified Collector 314 collects L_Model 270A, LR_Model270B, and/or Ci_Model 274 via a REST API, and the hardware information(e.g., configurations, tables, fabric card information, rules, routes,etc.) via SSH using utilities provided by the switch software, such asvirtual shell (VSH or VSHELL) for accessing the switch command-lineinterface (CLI) or VSH_LC shell for accessing runtime state of the linecard.

Unified Collector 314 can poll other information from Controllers 116,including, without limitation: topology information, tenantforwarding/routing information, tenant security policies, contracts,interface policies, physical domain or VMM domain information, GOB(out-of-band) management IP's of nodes in the fabric, etc.

Unified Collector 314 can also poll information from nodes (e.g., Leafs104 and Spines 102) in Fabric 120, including without limitation:Ci_Models 274 for VLANs, BDs, and security policies; Link LayerDiscovery Protocol (LLDP) connectivity information of nodes (e.g., Leafs104 and/or Spines 102); endpoint information from EPM/COOP; fabric cardinformation from Spines 102; routing information base (RIB) tables fromnodes in Fabric 120; forwarding information base (FIB) tables from nodesin Fabric 120; security group hardware tables (e.g., TCAM tables) fromnodes in Fabric 120; etc.

In some cases, Unified Collector 314 can obtain runtime state from thenetwork and incorporate runtime state information into L_Model 270Aand/or LR_Model 270B. Unified Collector 314 can also obtain multiplelogical models from Controllers 116 and generate a comprehensive ornetwork-wide logical model (e.g., L_Model 270A and/or LR_Model 270B)based on the logical models. Unified Collector 314 can compare logicalmodels from Controllers 116, resolve dependencies, remove redundancies,etc., and generate a single L_Model 270A and/or LR_Model 270B for theentire network or fabric.

Unified Collector 314 can collect the entire network state acrossControllers 116 and fabric nodes or members (e.g., Leafs 104 and/orSpines 102). For example, Unified Collector 314 can use a REST interfaceand an SSH interface to collect the network state. This informationcollected by Unified Collector 314 can include data relating to the linklayer, VLANs, BDs, VRFs, security policies, etc. The state informationcan be represented in LR_Model 270B, as previously mentioned. UnifiedCollector 314 can then publish the collected information and models toany downstream operators that are interested in or require suchinformation. Unified Collector 314 can publish information as it isreceived, such that data is streamed to the downstream operators.

Data collected by Unified Collector 314 can be compressed and sent todownstream services. In some examples, Unified Collector 314 can collectdata in an online fashion or real-time fashion, and send the datadownstream, as it is collected, for further analysis. In some examples,Unified Collector 314 can collect data in an offline fashion, andcompile the data for later analysis or transmission.

Assurance Appliance 300 can contact Controllers 116, Spines 102, Leafs104, and other nodes to collect various types of data. In somescenarios, Assurance Appliance 300 may experience a failure (e.g.,connectivity problem, hardware or software error, etc.) that prevents itfrom being able to collect data for a period of time. AssuranceAppliance 300 can handle such failures seamlessly, and generate eventsbased on such failures.

Switch Logical Policy Generator 316 can receive L_Model 270A and/orLR_Model 270B from Unified Collector 314 and calculate Li_Model 272 foreach network device i (e.g., switch i) in Fabric 120. For example,Switch Logical Policy Generator 316 can receive L_Model 270A and/orLR_Model 270B and generate Li_Model 272 by projecting a logical modelfor each individual node i (e.g., Spines 102 and/or Leafs 104) in Fabric120. Switch Logical Policy Generator 316 can generate Li_Model 272 foreach switch in Fabric 120, thus creating a switch logical model based onL_Model 270A and/or LR_Model 270B for each switch.

Each Li_Model 272 can represent L_Model 270A and/or LR_Model 270B asprojected or applied at the respective network device i (e.g., switch i)in Fabric 120. In some cases, Li_Model 272 can be normalized orformatted in a manner that is compatible with the respective networkdevice. For example, Li_Model 272 can be formatted in a manner that canbe read or executed by the respective network device. To illustrate,Li_Model 272 can included specific identifiers (e.g., hardware planeidentifiers used by Controllers 116 as classifiers, etc.) or tags (e.g.,policy group tags) that can be interpreted by the respective networkdevice. In some cases, Li_Model 272 can include JSON objects. Forexample, Li_Model 272 can include JSON objects to represent rules,filters, entries, scopes, etc.

The format used for Li_Model 272 can be the same as, or consistent with,the format of Ci_Model 274. For example, both Li_Model 272 and Ci_Model274 may be based on JSON objects. Similar or matching formats can enableLi_Model 272 and Ci_Model 274 to be compared for equivalence orcongruence. Such equivalency checks can aid in network analysis andassurance as further explained herein.

Switch Logical Configuration Generator 316 can also perform changeanalysis and generate lint events or records for problems discovered inL_Model 270A and/or LR_Model 270B. The lint events or records can beused to generate alerts for a user or network operator.

Policy Operator 318 can receive Ci_Model 274 and Hi_Model 276 for eachswitch from Unified Collector 314, and Li_Model 272 for each switch fromSwitch Logical Policy Generator 316, and perform assurance checks andanalysis (e.g., security adherence checks, TCAM utilization analysis,etc.) based on Ci_Model 274, Hi_Model 276, and Li_Model 272. PolicyOperator 318 can perform assurance checks on a switch-by-switch basis bycomparing one or more of the models.

Returning to Unified Collector 314, Unified Collector 314 can also sendL_Model 270A and/or LR_Model 270B to Routing Policy Parser 320, andCi_Model 274 and Hi_Model 276 to Routing Parser 326.

Routing Policy Parser 320 can receive L_Model 270A and/or LR_Model 270Band parse the model(s) for information that may be relevant todownstream operators, such as Endpoint Checker 322 and Tenant RoutingChecker 324. Similarly, Routing Parser 326 can receive Ci_Model 274 andHi_Model 276 and parse each model for information for downstreamoperators, Endpoint Checker 322 and Tenant Routing Checker 324.

After Ci_Model 274, Hi_Model 276, L_Model 270A and/or LR_Model 270B areparsed, Routing Policy Parser 320 and/or Routing Parser 326 can sendcleaned-up protocol buffers (Proto Buffs) to the downstream operators,Endpoint Checker 322 and Tenant Routing Checker 324. Endpoint Checker322 can then generate events related to Endpoint violations, such asduplicate IPs, APIPA, etc., and Tenant Routing Checker 324 can generateevents related to the deployment of BDs, VRFs, subnets, routing tableprefixes, etc.

FIG. 3C illustrates a schematic diagram of an example system for staticpolicy analysis in a network (e.g., Network Environment 100). StaticPolicy Analyzer 360 can perform assurance checks to detect configurationviolations, logical lint events, contradictory or conflicting policies,unused contracts, incomplete configurations, etc. Static Policy Analyzer360 can check the specification of the user's intent or intents inL_Model 270A to determine if any configurations in Controllers 116 areinconsistent with the specification of the user's intent or intents.

Static Policy Analyzer 360 can include one or more of the Operators 310executed or hosted in Assurance Appliance 300. However, in otherconfigurations, Static Policy Analyzer 360 can run one or more operatorsor engines that are separate from Operators 310 and/or AssuranceAppliance 300. For example, Static Policy Analyzer 360 can be a VM, acluster of VMs, or a collection of endpoints in a service functionchain.

Static Policy Analyzer 360 can receive as input L_Model 270A fromLogical Model Collection Process 366 and Rules 368 defined for eachfeature (e.g., object) in L_Model 270A. Rules 368 can be based onobjects, relationships, definitions, configurations, and any otherfeatures in MIM 200. Rules 368 can specify conditions, relationships,parameters, and/or any other information for identifying configurationviolations or issues.

Moreover, Rules 368 can include information for identifying syntacticviolations or issues. For example, Rules 368 can include one or morerules for performing syntactic checks. Syntactic checks can verify thatthe configuration of L_Model 270A is complete, and can help identifyconfigurations or rules that are not being used. Syntactic checks canalso verify that the configurations in the hierarchical MIM 200 arecomplete (have been defined) and identify any configurations that aredefined but not used. To illustrate, Rules 368 can specify that everytenant in L_Model 270A should have a context configured; every contractin L_Model 270A should specify a provider EPG and a consumer EPG; everycontract in L_Model 270A should specify a subject, filter, and/or port;etc.

Rules 368 can also include rules for performing semantic checks andidentifying semantic violations or issues. Semantic checks can checkconflicting rules or configurations. For example, Rule1 and Rule2 canhave aliasing issues, Rule1 can be more specific than Rule2 and therebycreate conflicts/issues, etc. Rules 368 can define conditions which mayresult in aliased rules, conflicting rules, etc. To illustrate, Rules368 can specify that an allow policy for a specific communicationbetween two objects can conflict with a deny policy for the samecommunication between two objects if the allow policy has a higherpriority than the deny policy, or a rule for an object renders anotherrule unnecessary.

Static Policy Analyzer 360 can apply Rules 368 to L_Model 270A to checkconfigurations in L_Model 270A and output Configuration Violation Events370 (e.g., alerts, logs, notifications, etc.) based on any issuesdetected. Configuration Violation Events 370 can include semantic orsemantic problems, such as incomplete configurations, conflictingconfigurations, aliased rules, unused configurations, errors, policyviolations, misconfigured objects, incomplete configurations, incorrectcontract scopes, improper object relationships, etc.

In some cases, Static Policy Analyzer 360 can iteratively traverse eachnode in a tree generated based on L_Model 270A and/or MIM 200, and applyRules 368 at each node in the tree to determine if any nodes yield aviolation (e.g., incomplete configuration, improper configuration,unused configuration, etc.). Static Policy Analyzer 360 can outputConfiguration Violation Events 370 when it detects any violations.

FIG. 4 illustrates an example configuration 400 for a network assuranceplatform 434. The network assurance platform 434 can run the assuranceoperator 402 on each Leaf 104 to generate and emit fault codes from theLeafs 104. In some cases, the assurance operator 402 can be, forexample, one or more operators from the operators 310 illustrated inFIG. 3A. The fault codes can represent errors, such as hardware errors.The assurance operators 402 can send the raw faults 404 to the logicalpolicy enrichers 406.

The logical policy enrichers 406 can map the hardware identifiers (e.g.,scope, pcTag, etc.) to the logical policy entity defined in the fabricconfiguration (e.g., ACI fabric configuration). For example, the logicalpolicy enrichers 406 can map the hardware identifiers to particulartenants, EPGs, application profiles (APs), contracts, etc. The logicalpolicy enrichers 406 can generate indexed faults 408 based on themappings, and send the indexed faults 408 to tenant aggregators 418,420, 422, 424. In some cases, the indexed faults 408 can be transmittedto the aggregators as pairs such as key and tag pairs. Each key canrepresent a specific dimension, such as a tenant, a contract, anapplication profile, and EPG pair, etc.

The aggregators 418, 420, 422, 424 can represent an aggregation layer.In some cases, the aggregators 418, 420, 422, 424 can be specificallyset to aggregate along a pre-determined dimension, such as tenant (e.g.,aggregator 418), contract (e.g., aggregator 420), application profile(e.g., aggregator 422), EPG pair (e.g., aggregator 424), etc. Theaggregators 418, 420, 422, 424 can generate faults along a specificdimension, such as faults by tenant 410, faults by contract 412, faultsby application profile 414, faults by EPG pair 416, etc.

The network assurance platform 434 can then generate and/or storevisualization data for specific dimensions. For example, the networkassurance platform 434 can maintain tenant error visualization 426,contract error visualization 428, application profile errorvisualization 430, EPG pair error visualization 432, and so forth. Thevisualizations can provide hardware-level visibility of errors alongspecific dimensions in an SDN, such as an ACI network. Moreover, thetenant error visualization 426, contract error visualization 428,application profile error visualization 430, EPG pair errorvisualization 432 can be stored in one or more respective storagelocations, such as databases or storage servers.

FIG. 5A illustrates an example flowchart for a network assurance model.At step 500, the method involves data collection. Data collection caninclude collection of data for operator intent, such as fabric data(e.g., topology, switch, interface policies, application policies,endpoint groups, etc.), network policies (e.g., BDs, VRFs, L2Outs,L3Outs, protocol configurations, etc.), security policies (e.g.,contracts, filters, etc.), service chaining policies, and so forth. Datacollection can also include data for the concrete, hardware model, suchas network configuration (e.g., RIB/FIB, VLAN, MAC, ISIS, DB, BGP, OSPF,ARP, VPC, LLDP, MTU, QoS, etc.), security policies (e.g., TCAM, ECMPtables, etc.), endpoint dynamics (e.g., EPM, COOP EP DB, etc.),statistics (e.g., TCAM rule hits, interface counters, bandwidth, etc.).

At step 502, the method can involve formal modeling and analysis. Formalmodeling and analysis can involve determining equivalency betweenlogical and hardware models, such as security policies between models,etc.

At step 504, the method can involve smart event generation. Smart eventscan be generated using deep object hierarchy for detailed analysis, suchas: Tenant, Leaf, VRFs, Rules; Filters, Routes, Prefixes, Port Numbers.

At step 506, the method can involve visualization. Formal models can beused to identify problems for analysis and debugging, in a user-friendlyGUI.

FIG. 5B illustrates an example method for fault code aggregation. Atstep 520, the assurance operators 402 obtain respective fault codescorresponding to one or more network devices in a network (e.g., Leafs104). At step 522, the logical policy enrichers 406 map the one or morenetwork devices and/or the respective fault codes to respective logicalpolicy entities defined in a logical policy model of the network, toyield fault code mappings. The logical policy model can be a model ofthe fabric and/or network generated based on the SDN or ACIconfigurations.

At step 524, the aggregators 418, 420, 422, 424 aggregate one or more ofthe fault code mappings along respective logical policy dimensions inthe network to yield an aggregation of fault codes across respectivelogical policy dimensions. At step 526, the network assurance platform434 presents, for each of the respective logical policy dimensions, oneor more hardware-level errors along the respective logical policydimension. In some cases, the network assurance platform 434 cangenerate visualization data or interface data for presenting the one ormore hardware-level errors along the respective logical policydimension. Such data can be based on the aggregation of fault codesacross the respective logical policy dimensions.

The disclosed technology addresses the need in the art for assuringtenant forwarding in a network environment. The present technologyinvolves system, methods, and computer-readable media for networkassurance in layer 1, layer 2 and layer 3 of the networked environment.The present technology also involves system, methods, andcomputer-readable media for network assurance for internal-internal(e.g., inter-fabric) forwarding and internal-external (e.g., outside thefabric) forwarding in the networked environment. In some examples, thenetwork assurances can be in non-fabric networked environments. Thenetwork assurance can be performed using logical configurations,software configurations and/or hardware configurations.

As discussed above, certain embodiments are directed to the creation ofpolicies that will define how Endpoints 122 can communicate with eachother via the Leafs 104 and the Spines 102 with Fabric 120. Embodimentsdisclosed herein include the creation and dissemination of five levelsof instruction information discussed above with respect to FIG. 2D thatwill configure Leafs 104 and Spines 102 to facilitate thatcommunication. Each level can include a change in content and/or format,although this need not always be the case.

By way of non-limiting example, FIG. 7 illustrates exampleconfigurations in networked environment 700. Tenant Configuration 740(e.g., an L_Model 270A configuration) can have VRF1 configured with BD1.BD1 can have one or more configured subnets (e.g., 10.1.*, 100.2.*,etc.). Subnets can be designated either private or public. In someexamples, subnets can have a default designation of private. In thisexample, subnet 100.2.* has been made public (e.g., accessible fromoutside Fabric 120 and Networked Environment 700). Subnet 100.2.* can beassociated with Web Server 746. For example, Web Server 746 can provide,via subnet 100.2.*, access to one or more web services (to devices)within fabric 120 and outside of fabric 120 via Network Device 114.

In order for communication to and from Web Server 746 to be facilitated,Tenant Configuration 740 must be sent from Controller 116 to the Leafs104 and Spines 102 of Fabric 120. Public subnet 100.2.*, must also beleaked to an external device (e.g., Network Device 114) from a borderleaf (e.g., Leaf N). For tenants to communicate outside the Fabric 120,the external network device (e.g., Network Device 114) must leak subnets(e.g., 175.*) to the border leaf (e.g., Leaf N). The leaked externalsubnets (e.g., learned routes) can then be propagated to the L3 tablesof the Leafs 104 and Spines 102 of the Fabric 120.

VRFs can operate in the Layer 3 domain and can enable multiple instancesof a routing table to coexist on the same router (e.g., Leafs 104 andSpines 102, etc.) at the same time. Since the routing instances areindependent, the same or overlapping IP addresses can be used acrossVRFs without conflict.

BDs can operate as a Layer 2 forwarding construct within the fabric andcan enable bidirectional flow of traffic between a Layer 2 bridgednetwork and Layer 3 routed network traffic. Each BD can reside in a VRF.Each BD can also define one or more subnets. IP address in the definedsubnets can belong to the specified BD.

A Logical Group (EPG) can be a managed object that is a named logicalentity and contains a collection of endpoints. Endpoints can be devicesthat are connected to the network directly or indirectly (e.g., servers,virtual machines, network-attached storage, etc.). Each EPG can beassigned to a BD. EPGs can have an address (e.g., identity), a location,attributes (e.g., version or patch level), and can be physical orvirtual. Endpoint membership in an EPG can be dynamic or static. Subnetscan also be defined at the EPG.

L3out can be a construct to extend Layer 3 for external communication(e.g., outside the fabric and/or network environment). L3out can bebound to a VRF. L2out can be a construct to extend the Layer 2 forexternal communication (e.g., outside the fabric and/or networkenvironment). L2out can be bound to a BD.

Controller 116, once configured, can push the L_Model 270A/B to theLeafs 104 and Spines 102 of the Fabric 120. In some cases, the L_Model270A/B can be pushed to all Leafs 104 and Spines 102 of the Fabric 120.In other cases, portions of the L_Model 270A/B can be pushed to theLeafs 104 and Spines 102 of the Fabric 120 that are responsible forrouting and forwarding data that are consistent with the L_Model 270A/B.For example, when BD1 defines subnet 10.1.1.*, the Leafs 104 that areconfigured to that BD (e.g., responsible for transmitting data withsubnet 10.1.1*) can receive the L_Model 270A/B configuration related toBD1.

The L_Model 270A/B can be used to populate forwarding and routing tables(L3), VLAN tables (L2), interfaces (L1) and endpoint table (L2 and L3)at Leafs 104 and Spines 102 of the Fabric 120. For example, when BD1defines subnet 10.1.1.*, the LPM table can populate an entry for thesubnet and a destination to which incoming packets are to be routed(e.g., Leaf 3).

In some examples, the LPM table can be part of the forwarding table. TheLPM table can specify one or more a subnets with a destination addressmatching more than one entry in the LPM table. The most specific of thematching table entries the one with the longest subnet mask can beselected. The LPM table can also include information relating tosubnets, static routes, dynamic (learned) routes, L3 interfaces, and L3loopbacks. Populated LPM tables in FIG. 7 include tables 742, 748, 1802and 1806, which are Ci_Models.

Each Leaf 104 in Fabric 120 can have a Layer3 (L3) Table (e.g., LongestPrefix match (LPM) Table). For example, L3 Table 742 can reside on Leaf2 and L3 Table 748 can reside on Leaf N. The L3 tables can include alisting of subnets and next hops. For example, when a tenant of Leaf Nrequests to access Web Server 746, Leaf N can look up in L3 Table 748the subnet associated with Web Server 746 (e.g., 100.2.*) and determinethe next hop (e.g., Leaf 2). Accordingly, Leaf N can direct the packetsfrom the tenant, to Leaf 2 via one of the Spines.

In other examples, devices outside Fabric 120 can request access topublic Web Server 746. In these examples, the border leaf (e.g., Leaf N)can leak the public subnet (e.g., 100.2.*) to the external networkdevice (e.g., Network Device 114). The external network device canmaintain routing and forwarding tables (e.g., Table 744) where theleaked subnet information is stored. When a device outside Fabric 120requests access to Web Server 120, Network Device 114 can determinewhere to forward the request by Table 744. The request will then berouted to the border leaf (e.g., Leaf N). The request can then be routedto the proper destination by the L3 tables of the Leafs and Spines.

In other examples, tenants within Fabric 120 can access devices externalto the fabric (through the border leaf to which the external network isconnected). For example, a tenant (e.g., EndPoint 4) can request accessto devices external to Fabric 120, such as subnet 175.*. L3 Table 742can determine that the next hop in accessing the external device is viaLeaf N (e.g., subnet 175.*→Leaf N). This learned route is leaked fromNetwork Device 114 to Border Leaf N, which is then propagated to theother leafs and spines of the fabric as needed. Border Leaf N can thenroute the request external to the fabric via next hop Network Device 114(e.g., from L3 Table 748).

Networked Environment 700 can also include routing and forwardingthrough security contracts (e.g., 750) between logical groups, Layer 3loopbacks and Layer 3 interfaces. While not shown, the L3 tables alsoinclude the Layer 3 loopback and Layer 3 interfaces.

The above instructions, and other instructions, are deployed by theControllers 116 as L_Models 270A/B and executed by the conversion intoCi_Model 274 and Hi_Model 276 as the individual Leafs 104 and Spines102. Network assurance is a process by which the various creation andpropagation of the L, C and H models, and related content, have beenproperly programmed, deployed, created etc. By way of non-limitingexample, single level checks can be performed at the logical, softwareand/or hardware models by verifying the consistency of the models.Checks can also be performed by checking whether models were properlyconverted into other levels, e.g., whether the Leafs 104 and Spines 102properly converted the L_Model 270A/B into Ci_Model 274 (logical toconcrete checks) and Ci_Model 274 into Hi_Model 276 (concrete tohardware checks).

Network assurance can be verified by Assurance Appliance 300. Appliance300 can poll devices (e.g., Spines 102, Leafs 104, Controller 116) ofthe networked environment (e.g., 100, Fabric 120) for the resident L, Hand C Model configurations. For example, the application can pollControllers 116 for L_Model 270A/B configurations and Leaf 104 andSpines 102 for Ci_Model 274 and Hi_Model 276 configurations. Afterreceiving one or more L, C and H Model configurations from the one ordevices (e.g., Spines 102, Leafs 104, Controller 116), Appliance 300 canperform one or more verifications using Checker 620 shown in FIG. 6 ,which can operate in the manner discussed with respect to FIG. 3B. It isto be understood that activity undertaken by one of Appliance 300 orChecker 620 can be undertaken by the other.

Checker 620 can be a component of Appliance 300. Checker 620 can receiveas input one or more configurations (e.g., received at Appliance 300through polling via Unified Collector 314), as well as other inputs asmay be provided by Network 100 and/or Fabric 120. In some examples, onemore L_Model 622 configurations (e.g., L-Model 270A/B) as provided byController 116 can be received as input. Similarly, reporting Leafs 104and Spines 102 provide, and Checker 620 receives as input, one or moreC_Model 624 configurations (e.g., Ci_Model 274) and one or more H_Model626 configurations (e.g., Ci_Model 276). Other data from components ofthe Fabric 120 and/or Network 100 may also be received at Checker 620.

One type of check is a single level consistency check, in which aparticular received model 622/624/626 is checked to confirm that theintent of the operator as originally programmed is represented andproperly configured in the model. For example, Checker 620 can performconsistency checks on the L_Model configuration 622. For example,performing syntactic analysis (e.g., configuration, setup,typographical, etc.) on the L_Model configuration 622. The consistencychecks can determine if the intent of the operator, when creating theconfiguration, is represented in the L_Model configuration 622.

A non-limiting example of a Layer 1 consistency check on L_Modelconfiguration 622 is EPG deployed on an ‘admin down’ interface. Thischeck can read the EPG deployment information and Interface Admin statusfrom the L-Model. The EPG deployment information can identify the Leafand interface where the EPG is physically deployed, and if properlydeployed would match the Leaf and interface in the Interface AdminStatus information. If the status is found to be ‘admin down’ the checkmay fail.

A non-limiting example of a Layer 2 check is overlapping encap VLANs ona switch. This check can read the EPG deployment information andInterface Port Policies from the L-Model to obtain the leaf, interfaceand encap VLAN. The EPG deployments can be grouped on a leaf by leafbasis. Within each such (leaf) group, the check can look for twodistinct EPGs having the same encap VLAN. When such a pair is found, thecheck may fail if the pair are on the same interface. In other examples,the check can search interfaces on which the matching EPG pair forInterface Port Policies information gathered from the L-model. The checkmay fail if the VLAN scope of both the ports is marked as global.

A non-limiting example of a Layer 3 check is overlapping subnets due tocross-VRF contracts. This check can read the BD (subnets and VRFassociated), EPG (subnets and BD association), and Contract informationfrom the L-model. Within the set of Contracts the check can search for aContract satisfying the cross VRF property as follows: VRF of BD ofprovider EPG of contract, namely provider VRF, and VRF of BD of consumerEPG of contract, namely consumer VRF are not the same VRFs. When such acontract is found, the check determines the consumer subnets (EPGsubnets as well as consumer EPG's BD subnets) and provider subnets (EPGsubnets). The checks may fail when any of the consumer subnets overlapswith any of the provider subnets.

Similarly, one or more C_Model 624 configurations (e.g., the Ci_Models274 from the reporting Leafs 104 and Spines 102) can be received as aninput and Checker 620 can perform consistency checks on any and allC_Model 624 configurations. Non-limiting examples of consistency checks,and how they are preformed, are as follows:

A non-limiting example of a Layer 1 check is Virtual Port Channel (VPC)consistency. The check can read the VPC state information from eachLeaf. The check may fail if some interfaces on one leaf are in VPC mode,but no interfaces on the VPC peer leaf are in VPC mode.

A non-limiting example of a Layer 2 check is EPG VLAN (FD-VLAN) and BDVLAN (BD-VLAN) relationship consistency. The check can read the VLANtable information of a Leaf. The check can also search an FD-VLAN, suchthat its parent BD-VLAN does not include in the list of its children—iffound, however, the check may fail. In other examples, the check mayfail if an identified FD-VLAN does not indicate the BD-VLAN as theparent.

A non-limiting example of a Layer 3 check is Learned route distributionand next hop consistency. The check can read the LPM table informationof all Leafs. The check may fail when a route is learned on some borderleaf via an L3out but the route is not present on all of the other leafswhere the L3out's VRF is deployed. In other examples, the check may failwhen the next hops of the route on non-border leafs does not contain theborder leafs.

There is no specific need for a separate H_Model consistency check, asthis would be encompassed by the consistency check of the correspondingC_Model; any absence of issues in the C_Model checks by extensionrepresents a lack of issues in the corresponding H_Model. However, theapplication is not so limited, and Checker 620 can perform an H_Modelconsistency check consistent with the methodologies discussed herein.

Another type of check is a model-to-model check. Such a check confirmsthat the Leafs 104 and Spines 102 properly converted an L_Model 270A/Binto a Ci_Model 274, and/or a Ci_Model 274 into an Ci_Model 276.

For an L-to-C model (“L-C”) check, a specific Leaf or Spine provides aCi_Model as C_Model 624 and Controller 116 provides an L_Model 622.Checker 620 can determine whether the specific Ci_Model was properlygenerated from the L_Model 622. Checker 620 first takes the L_Model 622and generates the corresponding local Li_Model for that particular Leafor Spine; this can be performed by Switch Logical Policy Generator 316as described with respect to FIG. 3B, for which the correspondinggenerated Li_Model is shown in FIG. 3 as Li. Since the particularCi_Model and the corresponding generated Li_Model are in differentformats, checker 620 converts one or both into the same format (“commonformat”). This format may be the native format of the Li_Model or theCi_Model (in which case only one of the models would need to undergoformat conversion), but can be a format different from both.Non-limiting examples of the common format is JSON objects, flattenedlist, strings, XML, common table, etc.

As discussed above, a particular Ci_Model 274 may have different contentthan the Li_Model 272 from which it was based, such that there will beoverlapping fields of content between the models. The content differencereflects in part that Li_Model represents instructions of operatorintent and the Ci_Model represents actual implementation of that intent,and thus the Ci_Model can include content regarding actualimplementation. Checker 620 can extract (at least some of the) contentfrom the overlapping fields from the common format version of theCi_Model and the corresponding generated Li_Model and compares themrelative to each other. If the extracted content from the overlappingfields matches, then the content was properly incorporated when thecorresponding Ci_Model was created. If the extracted content from theoverlapping fields do not match, then an error is generated, forexample, an error that prevented the proper creation or population ofthe corresponding Ci_Model. The error could have occurred in a number ofdifferent ways, for example, in the L-to-C conversion, or that theCi_Model is outdated (perhaps the updated L_Model never having beenreceived or converted at all); the application is not limited to thenature of the error.

For a C-to-H model (“C-H”) check, a specific Leaf or Spine provides itsspecific Ci_Model 624 and Hi_Model 626. Checker 620 can determinewhether the Hi_Model was properly generated from the Ci_Model. Since theparticular Ci_Model and the Hi_Model are in different formats, checker620 can convert one or both into a common format. This format may be thenative format of the Hi_Model or the Ci_Model (in which case only one ofthe models would need to undergo format conversion), but can be a formatdifferent from both. The common format of the C-H level check may be thesame or different from the format used in the L-C level checks.Non-limiting examples of the common format is JSON objects, flattenedlist, strings, XML, common table, etc.

As discussed above, a particular Ci_Model 274 may have different contentthan the corresponding Hi_Model 276 from which it was based, such thatthere will be overlapping fields of content between the modelsinformation. Checker 620 can extract (at least some of) the content fromthe overlapping fields from the common format version of the Ci_Model624 and the corresponding Hi_Model 626 and compares them relative toeach other. If the extracted content from the overlapping fieldsmatches, then the content was properly incorporated when thecorresponding Hi_Model was created. If the extracted content from theoverlapping fields does not match, then some error prevented the propercreation of the corresponding Hi_Model. The error could have occurred ina number of different ways, for example, in the C-to-H conversion, orthat the Hi_Model is outdated (perhaps the Leaf 104 simply neverperformed the conversion in the first place); the application is notlimited to the nature of the error.

In response to no discrepancies in the consistency or model-to-modelconversion check, Checker 620 can output an informational event 628. Inresponse to one or more discrepancies Checker 620 can output an errorevent 628 (e.g., of varying severity depending on the discrepancy).

FIG. 9 illustrates an example flowchart for an example network assuranceof model-to-model checks of a network environment. The method shown inFIG. 9 is provided by way of example, as there are a variety of ways tocarry out the method. Additionally, while the example method isillustrated with a particular order of sequences, those of ordinaryskill in the art will appreciate that FIG. 9 and the sequences showntherein can be executed in any order that accomplishes the technicaladvantages of the present disclosure and can include fewer or moresequences than illustrated.

Each sequence shown in FIG. 9 represents one or more processes, methodsor subroutines, carried out in the example method. The sequences shownin FIG. 9 can be implemented by an application in a network environmentsuch as network environment 100 shown in FIGS. 1A and 1B. Theapplication can be present on Appliance 300 and/or Checker 620, althoughthe application is not so limited.

At sequence 910, the various models (e.g., L_Model, C_Model, and/orH_Model) to be checked are received. For example, each Controller 116 inthe Fabric 120 can send to the application (e.g., Assurance Appliance300 and/or Checker 620) one or more L_Models 270A/B, and each Leaf 104and Spine 102 sends its corresponding Ci_Model 274, and Ci_Model 276. Inanother example, some but not all of the L_Model, C_Model, and/orH_Model configurations are provided and/or received. Those receivedmodels will include information content, at least some of which will besubject to the validation process.

At sequence 920, the application can convert two received models into acommon format (i.e., one model is converted into the format of the otherso they are the same, or both models are converted into the same newformat). In some examples, the common format can be the common model(e.g., that the configuration was original entered in by the operator).

At sequence 930, the application can validate the received informationby comparing content from relevant overlapping fields in the commonformat of the two models. A match in content represents that, at leastfor that content, the originating Leaf 104 or Spine 102 properlyconverted one level of model into the next level of model. For example,the logical model (e.g., L_Model) received from the controller wasproperly converted into the concrete model (e.g., C_Model). Adiscrepancy indicates some error in relation to the deployment orcreation of the models.

At sequence 940, an event is generated. For example, when thevalidations are consistent the application can generate an informationevent. In other examples, when one or more of the validations are notconsistent the application can generate an error event. The severity ofthe error event can at least be determined based on which validation isinconsistent and which device had the inconsistency (e.g., productiondevice, test device, access point, border leaf, etc.).

The above methodology reflects the overall steps for model-to-modelchecks. Individual types of model-to-model checks may include additionalsteps, modify certain steps, or omit certain steps. Non-limitingexamples of such individual checks are set forth in other sectionsherein.

In the above embodiments, the local Li_Model 272 is not requested orprovided to Checker 620, as the configuration and/or level checks thatinclude the L_Model 270A/B would encompass any specific checks on localLi_Model 272. However, the application is not so limited, and a localLi_Model 272 can be an input to Checker 620 and checked consistent withthe methodologies discussed above.

In the above embodiments, there is no need for a separate L-to-H model(“L-H”) check, as this would be encompassed by the combined results inthe L-C and C-H checks; the absence of discrepancies in the L-C and C-Hchecks by extension represents a lack of discrepancies between L and H.However, the application is not so limited, and Checker 620 can performan L-H level check consistent with the methodologies discussed above.

Consistency checks may be universal to an entire model, in that allcontent of the model is checked. In other examples, only some content ofthe model may be checked, and in particular topic specific content. Byway of non-limiting example, a Layer 2 specific check may only examinecontent in the model that is specific to Layer 2 operations, while aLayer 3 specific check examines content in the model that is specific toLayer 3 operations.

In some examples, model-to-model checks may be universal to all contentof the model, in that all content (at least all content found inoverlapping fields of the models that are to be compared) can becompared to identify which specific fields have matches ordiscrepancies. To the extent that certain specific topic checks are ofinterest, the check can simply consult the resulting overall comparisonto examine the relevant fields. By way of a non-limiting example, the Land the C level may have 50 overlapping fields of content, but a firstparticular check may only require consideration of three (3) of them,while a second particular check only requires consideration of two (2)of them. Each topic specific check can consult the results of thecomparison search for the needed specific fields. In this way, thecomparison can precede the request for a topic specific check.

In other examples, some content of the models may be compared, and inparticular topic specific content. For example, as discussed in moredetail with respect to individual checks below, some checks are targetedto specific information with respect to a specific topic. In such cases,the comparison may not include or consider all overlapping fields, andwould instead be limited to the overlapping fields of interest. By wayof non-limiting example, a Layer 2 specific check may only comparecontent of overlapping fields in specific to Layer 2 operations. Thetypes of information of interest relative to a particular check can bepredefined in the application (e.g., 300), although the application isnot so limited. In this way, the request for a topic specific check canprecede the comparison, and the comparison can be limited only to thetopics of interest.

Topic specific checks may be self-contained or overlap. Specifically, asdiscussed above different topic checks may examine different overlappingfields within the models. Some of these fields may be common todifferent checks. By way of a non-limiting example and as discussedbelow, Layer 2 checks for both BD deployment and an EPG deployment, bothof which examine the information about the VLAN; thus two differentchecks include consideration of the similar content. If the checks areself contained, then each check would examine the relevant contentregardless of other prior/future checks; thus in the example the VLANwould be checked independently in both the BD deployment and an EPGdeployment check. In the other examples, if a common field has alreadyby examined by a prior check, then a subsequent check can rely uponthose results rather than conduct an independent examination; thus ifthe BD check shows that the VLAN information is as expected, then asubsequent EPG check can adopt those results rather thanagain/separately examine the VLAN information.

In the above embodiments, the results of the check are either a match ora discrepancy. However, the application is not so limited. There can bea predefined range of acceptable variation between compared content asto be acceptable or otherwise consistent with expectations. There couldbe some content for which a particular discrepancy is less importantthat another type of discrepancy. The application is not limited to themanner in which Checker 620 evaluates the existence, nature or impact ofdiscrepancies.

Although the above checks are discussed with respect to specific Leafs104 and Spines 102, it is to be understood that these checks can beperformed on all or some Leafs 104 and Spines 102. Such checks could bein response to simply receiving the models, e.g., a C_Model and H_Modelare received, and thus a C-H check is performed. In other examples,specific Leafs 104 and/or Spine 102 could be polled for information andthe check confined to request information. The application is notlimited to the manner in which information for checks are requested orreceived, or what triggers Checker 620 to institute a particular check.

Tenant Forwarding Event Generation

As shown above, FIG. 6 also illustrates the generation of an Event 628.Events can be generated in response to Checker 620 performing a check onone or more inputs (e.g., 622, 624, 626, 630). Events can beinformational events and error events. Informational events aregenerated in response to no inconsistencies or discrepancies foundduring the check of the inputs. The information event can include thecheck performed, the devices checks, the time checked, the result of thecheck, etc. The informational (and error) events can be stored in amemory/database accessible by Appliance 300.

Historically stored events can be used in a predictive manner. By way ofnon-limiting example, the cause and effect (and particularly thefailure) of prior configurations and checks of those configurations canbe stored and referred to. Before changes are made to the Fabric 120 ornetworked environment with a newly proposed configuration, the systemcan consult with the historical data for instances of the same orsimilar configurations and any resulting positive or adverse effects ofthe same. The system can thus predict if the newly proposedconfiguration will adversely affect the network based on previousconfigurations that are the same or similar in nature. Similarly, thehistorical events can also be used to predict changes that have recentlytaken affect that could cause issues.

The error events can be generated in response to inconsistencies ordiscrepancies found during the check of the inputs (e.g., L_Model,C_Model, H_Model, etc.). The error events can have different levels ofseverity, for example, warning, minor, major, critical, etc. A warningseverity can be a potential problem in the future, such as, not usingbest practices, or events that are of interest to the operator. A minorseverity can be a low impact problem, for example, does not affectproduction devices, operation not affected, etc. A major can be a highimpact problem, for example, operation is affected, but is stilloperational, (e.g., production devices dropping some packets, etc.). Acritical error is a serious failure that usually prevents normaloperation of devices and requires an immediate fix, for example, failureof border leaf, production devices, etc.

Each event can include a mnemonic (event code), a description of theerror, remediation steps, affected objects (e.g., VRF), secondaryaffected objects (e.g., BD and EPG linked to affected VRF), severitylevel, number of checks, etc. When an error has occurred with anaffected object, each secondary affected object can be checked forerrors. For example, when a VRF has an issue and an error event isgenerated, the secondary objects of the VRF (linked BDs, EPGs, etc.) canbe checked for errors. In some examples, the error might be generated ata higher level (e.g., VRF), but the cause of the error could be from alower level (e.g., BD, EPG, etc.). By checking secondary objects anddetermining the actual cause of the error, remediation can be moreefficient and effective.

In some examples, the remediation steps can be transmitted to theoperator. In other examples, one or more of the devices in the networkenvironment or fabric can automatically perform the remediation steps.In some examples, Appliance 300 can perform the remediation steps. Inother examples, the remediation can require a vendor and the operatorcan be notified to contact the vendor.

Validation of Layer 1

From the perspective of deployment the operator can define in the LModel 270A instructions as to particular Leafs 104 or Spines 102regarding Layer 1 operability. This can include whether a physical link(e.g., port) is set to active or down, a software link (e.g., port) isactive or down, and a configured check to see if the interface isconfigured or not configured (e.g., access policy configured, etc.).Leafs 104 and Spines 102 can receive the L_Model 270A, convert therelevant content into Ci_Model 274, and then execute on the instructionsto institute the configurations and set the physical link, softwarelinks and interfaces in the manner instructed. Leafs 104 and Spines 102then update their respective Ci_Models 274 to reflect the status of thephysical, software links, interfaces and any combination thereof.

Network assurance of Layer 1 physical interfaces is directed toAppliance 300 and/or Checker 620 confirming that the configurations arepresent and the links are set as instructed. The Layer 1 checks can be aphysical link check (e.g., active or down), software link check (e.g.,active or down) and a configured check (e.g., configured or notconfigured). For Layer 1 to be actively configured, each of the threecontrols should be configured properly (e.g., physical link up, softwarelink up, access policy configured). In other examples, Layer 1 may notbe actively configured and the three controls can be up, down and notconfigured.

The physical link check can determine if the network interfaces (e.g. ofLeafs 104 and Spines 102) have a physical connection (e.g., cableattached, etc.). The Checker 620 receives the Ci_Model from a particularindividual Leaf 104 or Spine 102, and identifies in the Ci_Model thereported status of whether the physical link is active or down. Checker620 can also poll the software of the particular individual Leaf orSpine that monitors the hardware status to determine if the actual stateof the physical link is as reflected by the Ci_Model, such as by issuingcommands on the command line interface of a device (e.g., ethtool,ifconfig, etc.). The physical link is validated when the states match,and a discrepancy exists when the states do not (e.g., the polled statesays the link is active but the reported state shows it is down, orvice-versa).

The software link check can determine when the physical link (e.g.,port) is up or down at the software level (e.g., firmware, application,etc.). The methodology tracks that of the physical link check above,save that Checker 620 polls the software of the particular individualLeaf or Spine that monitors the software status to see if the actualstate of the software link is as reflected by the Ci_Model, such asports of a device having been brought up or down through software (e.g.,a command line interface (or GUI) of the device). The software link isvalidated if the states match, and a discrepancy exists if they do not(e.g., the polled state says the link is active but the reported stateshows it is down, or vice-versa).

The configured check can determine when the interface is configured(e.g., has an access policy, etc.). Access policies can govern theoperation of switch access ports that provide connectivity to resourcessuch as storage, compute, Layer 2 and Layer 3 (bridged and routed)connectivity, virtual machine hypervisors, Layer 4 to Layer 7 devices,etc. Access policies can enable the configuration of, but not limitedto: port channels and virtual port channels, protocols such as LLDP,CDP, or LACP, speed, STP properties, and features such as monitoring ordiagnostics. Access policies can include common objects including, butnot limited to: leaf profiles, leaf policies, interface profiles,interface policies, AEP, VLAN pools, physical domain (physdom), VMMdomain, hypervisor, etc. Referring now to FIG. 3D, the particular checkis an L-Model level consistency check, for which checker 620 checks toconfirm that the switch profile 382 and interface profile 384 arepresent, populated and properly linked 386, and that the interfaceprofile 384 is properly linked 386 to the global profiles 388. Checks ofglobal profiles and other content can occur at other layers.

FIG. 8 illustrates an example flowchart for an example network assuranceof Layer 1 physical interfaces. The method shown in FIG. 8 is providedby way of example, as there are a variety of ways to carry out themethod. Additionally, while the example method is illustrated with aparticular order of sequences, those of ordinary skill in the art willappreciate that FIG. 8 and the sequences shown therein can be executedin any order that accomplishes the technical advantages of the presentdisclosure and can include fewer or more sequences than illustrated.

Each sequence shown in FIG. 8 represents one or more processes, methodsor subroutines, carried out in the example method. The sequences shownin FIG. 8 can be implemented in a network environment such as networkenvironment 100 shown in FIGS. 1A and 1B. The application can be presenton Appliance 300 and/or Checker 620, although the application is not solimited.

At sequence 810, the various models (e.g., L_Model, C_Model, and/orH_Model) to be checked are received. For example, one or more Leafs 104,Spines 102 and Controllers 116 in Fabric 120 can send to the application(e.g., Assurance Appliance 300) L_Model 270A/B, Ci_Model 274, andH_Model 274. Those received models will include information content, atleast some of which will be subject to the validation process. Forexample, Layer 1 content of interest can include the physical linkstatus of the interfaces of the devices, the software link status of theinterfaces on the devices, and any access policies for the interfaces onthe devices.

At sequence 820, the application can validate the received information.For example, the application (e.g., Assurance Appliance 300) canvalidate the received information verse previously stored information.In some examples, the received physical interface status can bevalidated against previously stored information required to determine ifthe physical interface is up (e.g., active). In some examples, thereceived software interface status can be validated against previouslystored information required to determine if the software interface is up(e.g., active). In some examples, the received access policy informationcan be validated against previously stored information required todetermine if the access policy is present and properly configured. Insome examples, checker 620 can independently obtain the information fromthe one or more Leafs 104, Spines 102, etc., and compared against thecontent of the models in the manner discussed above.

At sequence 830, the application can generate an event. For example,when the validations are all successful (e.g., consistent withexpectations of the validation) the application can generate aninformation event. In other examples, when one or more of thevalidations are not successful (e.g., not consistent with expectationsof the validation) the application can generate an error event. Theseverity of the error event can at least be determined based on whichvalidation is unsuccessful and which device was unsuccessfully validated(e.g., production device, test device, access point, border leaf, etc.).

Validation of Layer 2

Layer 2 is the protocol layer (e.g., data link layer) that can providethe functional and procedural configurations to transfer data betweennetwork entities. Appliance 300 can perform a series of networkassurance checks on Layer 2 of the network environment. Layer 2 checkscan include at least, for Layer 2 relevant content, a consistency checkof the L_Model, an L-C check, and a C-H check, although the applicationis not so limited. These checks can be performed by Appliance 300 and/orChecker 620 as described above on at least the overlapping fields ofcontent that are specific to Layer 2 operability.

The Layer 2 checks can be performed on Access Policies (e.g., commonobjects) and tenant trees (e.g., common objects that are pushed to leafsand spines). The tenant tree common objects can include, for eachtenant, VRF, BD (bound to VRF), EPG (bound to BD). Each EPG can includedeployment information (where the controller should deploy theconfiguration), including static binding or dynamic binding and domains.The static binding can include the leaf, card, port, and VLAN. Thedynamic binding does not require further information as it is configureddynamically when required. The domains can include physical, VMM, orboth. The domains and static bindings can also include deploymentsettings including pre-provision, immediate, lazy, etc. Once theconfigurations are completed (e.g., by the operator) to define theL_Model 270A, the Controller 116 can push the L_Model 270A/B to theappropriate Leafs 104 and Spines 102. Appliance 300 and/or Checker 620can then check the deployment.

The Layer 2 L_Model consistency check can include at least a syntacticanalysis (e.g., configuration, setup, typographical, etc.) on theL_Model configuration. Specific content checked for Layer 2 includes byway of non-limiting example that the Access Policies are configuredproperly (e.g., based on the common objects), the EPG is configuredproperly (e.g., based on common objects), Cross EPG checks (e.g., if anEPG is using one VLAN another EPG deployed on the same Leaf should notbe using the same VLAN), etc. When there are inconsistencies an eventcan be generated.

The Layer 2 L-C checks can include if the BD is properly deployed (e.g.,semantic check). For example, some subset of BDs has been deployed onone or more of the Leafs 104. The L-C checks can validate the subset ofBDs have been properly deployed on the Leafs, for example, by utilizingthe common objects (e.g., VLAN and interface). In some examples, theVLAN and interface information (e.g., C_Model) can be stored in the VLANtable of the leaf/spine.

The L-C checks can also include if EPGs are properly deployed, whichsimilar to the BD check above examines the VLAN pair (between L_Modeland C_Model) and interface pair (between L_Model and C_Model).

The Layer 2 C-H checks can be similar to the Layer 2 L-C checks,although content to be checked resides and is received from the Leafs104 and Spines 102. For example, the application (e.g., Appliance 300)can receive the Ci_Model 624 and Hi_Model 626 from the one or more Leafs104 and Spines 102, convert to a common format and then validate theVLAN pairs and interface pairs. When there are inconsistencies betweenthe Ci_Model and Hi_Model an event can be generated.

Another Layer 2 check is to validate VLAN assignments in the network,which is an L_Model to H_Model relative check. In some examples, theL-Model to H_Model check can be performed by a combination of the L-Cand C-H checks. The checks can verify for each EPG, the same VLAN isassigned at the same interface at the L_Model, Ci_Model and Hi_Model.

Another Layer 2 check can be to validate VLAN assignments across thenetworked environment. This particular check is a comparison of contentof the L_Model to a Distributed Virtual Switch (DVS) configuration(hereinafter “DVS_Model”). The DVS is outside of Fabric 120 and likeLeafs 104 and Spines 102 also receives the L_Model 270A/B. Checker 620receives the L_Model 270A/B from Controller 116 L_Model input 622 asdiscussed above, and also receives as a distinct input the DVSconfiguration from the DVS at Other input 630. One or both of thereceived L_Model and/or the DVS configuration are converted into acommon format, and content from relevant overlapping fields (within themodels/configurations) is compared. For VLAN assignments, relevantoverlapping fields can include but are not limited to: the configuredVLANs for each EPG (e.g., static/physical and dynamic/virtual), and theparticular VLAN IDs for dynamic/virtual EPGs.

Another Layer 2 check is VLAN provisioning on a switch or port as partof the L_Model consistency check. Appliance 300 can check that theL_Model 270A/B received from the Controller 116 does not includeoverlapping VLANs. At the switch level, the Appliance 300 can determineif there are duplicate VLANs on the switch. At the port level, theAppliance 300 can determine if there are duplicate VLANs on the sameports of the switch. In the port example, there can be duplicate VLANson different ports of the switch.

Execution of the L-DVS, L-C and C-H level checks in Layer 2 occur asdiscussed with respect to FIG. 9 . The various received models atsequence 910 would include information relevant to the Layer 2 analysis,such as VLAN and interface information. At sequence 920, for eachmodel-to-model check the application can convert the received modelsinto a common format. At sequence 930, the application can validate thereceived information by examining content from relevant fields, such asthe VLAN common object and/or the interface common object of theCi_Model. Other common objects for Layer 2 can be validated betweenL-DVS, L-C and C-H checks.

At sequence 940, the controller can generate an event. For example, whenthe validations are consistent (e.g., consistent with expectations ofthe validation) the application can generate an information event. Inother examples, when one or more of the validations are not consistent(e.g., inconsistent with expectations of the validation) the applicationcan generate an error event. The severity of the error event can atleast be determined based on which validation is inconsistent and whichdevice had the inconsistency (e.g., production device, test device,access point, border leaf, etc.).

Validation of Layer 3

Appliance 300 can perform a series of checks on Layer 3 (e.g., networklayer) of the network environment (e.g., network assurance). The networklayer is responsible for packet forwarding including routing throughintermediate routers. Layer 3 checks can include at least a consistencycheck of the L_Model, an L-C check, and a C-H check, although theapplication is not so limited. These checks are performed by Appliance300 and/or Checker 620 as described above on at least the overlappingfields of content that are specific to Layer 3 operability.

There are a variety of Layer 3 checks that can be performed, forexample, BD subnets are deployed correctly (e.g., routing on/off,IP/mask, etc.), overlapping subnet checks, L3out deployed properly,BD-L3out association, RIB is programmed properly with subnets, learnedroutes are properly propagated in the fabric, RIB-FIB equivalence, andCross-VRF route leaking.

In Layer 3, a concern of the check can be integrity of deployment of thesubnets. Each BD can define one or more subnets (e.g., 10.0.*). Thesubnets can also be defined in the Logical Groups (e.g., EPG). Each VRF(which is in Layer 3) can have a unique IP address. If another deviceuses the same IP address, there is not a conflict as long as the IPaddress is used a different VRF. In Layer 3, a common object, L3out canalso be defined (for external communication). The L3out common objectcan define: L3 interfaces, L3out EPG, static routes and loopbacks.

Whether the BD subnets are deployed correctly can be an L-C check. Thesubnets for each BD (and EPG) can be resident on L_Model 270A/B deployedby the Controller 116. The subnets as deployed at the network devicescan include a routing information base resident on the Ci_Model of thereporting Leafs 104 or Spines 102. The application can collect the BDsubnets deployed from the received L_Model and Ci_Models based on VRFassignments to define a set union (e.g., a list of BD subnet deployedwithout duplicates). The application can determine that the BD subnetsare actually deployed at the appropriate network devices in the union(and not network devices that they should not be deployed). The BDsubnets check can also include checking the BD subnets on networkdevices because of contracts between EPGs. The BD subnet check candetermine the BD subnets points to the spine proxy and, the BD subnetshave a corresponding gateway address is installed.

FIG. 10 illustrates an example flowchart for an example networkassurance of Layer 3 configuration of a network environment. The methodshown in FIG. 10 is provided by way of example, as there are a varietyof ways to carry out the method. Additionally, while the example methodis illustrated with a particular order of sequences, those of ordinaryskill in the art will appreciate that FIG. 10 and the sequences showntherein can be executed in any order that accomplishes the technicaladvantages of the present disclosure and can include fewer or moresequences than illustrated.

FIG. 10 represents one or more processes, methods or subroutines,carried out in the example method. The sequences shown in FIG. 10 can beimplemented in a network environment such as network environment 100shown in FIGS. 1A and 1B. The application can be present on Appliance300 and/or Checker 620, although the application is not so limited.

Method 1000 can begin at sequence 1010. At sequence 1010 the variousmodels (e.g., L_Model, C_Model, and/or H_Model) to be checked arereceived. For example, the one or more Leafs 104, Spines 102 andControllers 116 in the Fabric 120 can send to the application (e.g.,Assurance Appliance 300) L_Model 270A/B, Ci_Model 274, and Ci_Model 276.Those received models include information content that can be subject tothe validation process. In some examples, the information relevant to aLayer 3 check can include prefix, subnet, network, or routing data fromeach VRF, BD, EPG (e.g., logical group), in the network environment orfabric.

At sequence 1020, one or more VRF containers for configuration abc(e.g., individual or multiple tables, files or the like) are created andpopulated for the VRFs in the received L_Model 622. FIG. 11 illustratesexample VRF containers 1100 and 1125. While VRF containers 1100 and 1125illustrate two different container configurations, more containerconfigurations are realized and the example containers 1100 and 1125 arenot limiting. Specific Leafs 104 under each VRF can be identified (e.g.,each VRF will have one or more BD subnets, and each BD subnet will haveone or more EPGs with one or more specific Leafs 104). Example VRFcontainer 1100 and 1125 can populated with at least the receivedCi_Model 274 and the created Li_Model 272 (as previously or newlycreated, and accounting for security contracts as discussed below) forthe specific Leafs. Each VRF container can thus include data specific tothat VRF (e.g., LPM container of prefixes, subnets, networks, externalroutes, etc.).

At sequence 1030, each VRF container is validated by Checker 620 and/orAppliance 300. The Li_Models represent how the BD subnets are to beconfigured (e.g., per the operator intent), and the Ci_Models representhow the instructions were actually implemented (e.g., at the Leaf,Spine, etc.). The intent should match the implementation, and thusChecker 620 confirms for each VRF container that the Li_Models areconsistent with the Ci_Models. A discrepancy exists if there is amismatch in content.

A discrepancy can also exist if a model is present that should not be.For example, if only Leafs 1 and 2 are under a particular VRF, thenLi_Model, Ci_Model, L2_Model and C2_Model models would be present in thecontainer for that VRF. No other Li_Model or Ci_Model for another Leafshould be in the VRF container, and the presence of such a model wouldbe a discrepancy.

FIG. 11 illustrates a deployment 1150. For example, as shown in FIG. 11, abc is on Leaf 1 and Leaf 2, but not on Leaf 3, while def is on Leaf3, but not Leafs 1 or 2. As such, Checker 620 will validate the abcconfiguration is properly deployed on Leaf 1 and Leaf 2, and not on Leaf3, and that the def configuration is properly deployed on Leaf 3 but notLeafs 1 or 2. Checker 620 can also validate the abc configuration onLeaf 1 and Leaf 2 is consistent with the abc configuration as defined inthe L_Model, and that def configuration on Leaf 3 is consistent with thedef configuration as defined in the L_Model.

In some examples, Checker 620 can perform set checks. For example, thechecker can perform a set difference by which after conversion to commonformat the Li_Model set subtracted from the Ci_Model set, and theCi_Model set is subtracted from the L_Model set; the sets are properwhen then the results of both subtractions are the same. A mismatchwould indicate missing or extra information that can result in, forexample, lost packets in the fabric and extra information can create,for example, security issues in the fabric (where device can accessother devices they do not have permission to ask).

While the example shows an L-C check, a C-H check can be realized. Insome examples, after the checker has validated the configuration thespine proxies can be checked to determine the correct next hop.

At sequence 1040, one or more events can be generated. For example, whenthe validations are consistent (e.g., consistent with expectations ofthe validation) the application can generate an information event. Inother examples, when one or more of the validations are not consistent(e.g., inconsistent with expectations of the validation), configurationsare not on the correct leafs (according the L_Model) or when there aremissing or extra information the application can generate an errorevent. The severity of the error event can at least be determined basedon which validation is inconsistent and which device had theinconsistency (e.g., production device, test device, access point,border leaf, etc.).

The above VRF container check of BD deployment is an example of anapplication of a generic VRF container integrity check. When the VRFcontainer is created, it can be populated with the L_Model, Li_Model,Ci_Model and/or Hi_Model that contain data relevant to/required by thecheck. Checker 620 can confirm the presence or absence of these models.By way of non-limiting examples, an appropriate Li_Model being presentbut a corresponding required Ci_Model not being present, would indicatean error in deployment. A Ci_Model being present but a requiredcorresponding Li_Model not being present, would indicate improper extrainformation. An Li_Model or Ci_Model being present from another VRF,would also indicate improper extra information.

Security Contract Between Logical Groups

In some cases, there can be a security contract between endpoint groups(as illustrated below). In these cases the leafs will get the leakedsubnets (and other information). Appliance 300 can perform an L-C checkto determine if the RIB is properly programed. For example, asillustrated in FIG. 12 , when Contract 2 is configured, the RIB on Leaf1 will include subnets 100.* and 200.* (e.g., from BD1) and subnets ofBD2 (e.g., 300.*, 400.*) leaked from Contract 2. Also, Leaf 3 willinclude subnets 300.* and 400.* (from BD2) and subnets for BD1 (e.g.,100.* and 200.*) leaked from Contract 2. Without Contract 2 in place,Leaf 1 will not include BD2 subnets and Leaf 3 will however include BD1subnets because of Contract 1.

Logical Groups (e.g., EPGs) cannot be properly deployed if there is aclash between two or more Logical Groups. For example, if Logical Groupsare using same interface, subnets, encap VLANs etc. In one example, asshown in FIG. 7 , Logical Group 1 (LG1) and Logical Group 2 (LG2) areinitially independent from each other and do not clash. However, whenSecurity Contract 750 is implemented, a clash between LG1 and LG2subnets is possible. Subsequent a security contract being formed, theLogical Groups can be deployed on the leaf that hosts the Logical Groups(e.g., endpoints within the logical groups can communicate). After asecurity is formed between Logical Groups, the Logical Groups can bedeployed on the leafs that host the Logical Groups and the leafs thathost a Logical Group which is an endpoint to the contract (e.g.,endpoints between logical groups that have a contract cancommunication). This enables the end points in the different LogicalGroups to communicate with one another. For example, LG1 is on Leaf 1and LG2 is on Leaf 2. After a security contract is formed between LG1and LG2, LG1 subnets will be deployed on Leaf 2 and LG2 subnets will bedeployed on Leaf 1. However, the communication between Logical Groupsthat have a contract can also share prefixes (e.g., routes) between theleafs, which can result in a conflict because of the existence of acommon prefix between the logical groups; when the configurations of LG1and LG2 would clash (e.g., same subnet), the deployment might fail andan event generated.

In some examples, the contract can be between Logical Groups that residein different VRFs. In this situation, internal routing leaking can beperformed between the VRFs, and the checks can be in Layer 3, discussedbelow.

FIG. 12 illustrates a graphical view of a tenant configuration 1200 of anetwork environment (e.g., Fabric 120). Tenant configuration 1200 canillustrate two BDs (e.g., BD1 and BD2) bound to the same VRF (e.g.,VRF1). Each BD can have one or more assigned subnets. For example, BD1can have assigned subnets 100.* and 200.*, and BD2 can have assignedsubnets 300.*. One or more Logical Groups (e.g., EPGs) can be bound tothe one or more BDs. For example, EPG1 and EPG2 can be bound to BD1 andEPG 3 can be bound to BD2. Each EPG can have one or more assignedsubnets as well. For example, EPG3 has assigned subnet 400.*. Each EPGcan point to one or more Leafs. For example, EPG1 can point to Leaf 1and Leaf 2, EPG2 can point to Leaf 2 and EPG3 can point to Leaf 3.

In the above tenant configuration (e.g., 1200), the endpoints within anEPG can communicate with one another, however, endpoints of one EPGcannot communication with endpoints of a different EPG without asecurity contract. For example, endpoints of EPG3 cannot communicationwith endpoints of EPG2 without first having a security contract (e.g.,Contract 1). When contract is in place between EPGs, endpoints of theEPGs subject to the contract can communicate. For example, when Contract1 is in place, the endpoints in EPG3 and EPG2 can communicate.

When a security contract exists, the configurations (and subnets) willbe deployed in the respectively connected Leafs. For example, whenContract 1 is in place, EPG2 subnets will be deployed on Leaf 3 and EPG3subnets will be deployed on Leaf 2. As such, the subnets of BD1 (e.g.,100.* and 200.*) will be deployed on Leaf 3 and the subnets of BD2(e.g., 300.*) and EPG3 (e.g., 400.*) will be deployed on Leaf 2.

When subnets are deployed on leafs, Appliance 300 can verify the subnetsare properly deployed, for example, by looking at resultingconfiguration created by the security contracts as an L-C check. Thesubnets can be verified by the security contracts and can create a VRFcontainer (e.g., 1250), as previously discussed. VRF container 1250, caninclude the Li_Model configurations including the configurations createdby security contracts. For example, Leaf 1-3 can all include subnets100.*, 200.* and 300.*. Subnet 300.* is accessible by Leaf 1 and 2 basedon Contract 1 and subnet 300.* is a BD subnet. Leaf 2 and 3 can bothhave subnet 400.* accessible based on Contract 1. The Li_Modelconfiguration can then be mapped to a Ci_Model configuration via the L-Ccheck to determine consistency across the fabric.

Next Hop Check

Next, the application can determine if the next hops are proper. Forexample, each BD and EPG subnet should point to a spine proxy and thegateway IP should point to local. In some examples, route table 1275 canshow the next hops for IP address and masks (e.g., using Longest PrefixMatch). When an IP address of 100.0.0.1 comes in, the next hop will beNext Hop 1 because the full IP address is used (e.g., 32 bit mask inIPv4). When an IP address of 100.0.0.0/24 comes in, the next hop will beNext Hop 2 because mask of 28 (also shown as a *). This check can be aconsistency check at the Ci_Model for the noted content.

RIB to FIB Equivalence

Appliance 300 can also determine that the RIB and FIB of a particularLeaf 104 or Spine 102 are consistent. The Routing Information Base (RIB)is contained within the LPM table of the Ci_Model of the particular Leaf104 or Spine 102. The Forwarding Information Base (FIB) is containedwithin the LPM table of the Hi_Model of the particular Leaf 104 or Spine102. The RIB to FIB check is a C-H check as described above on at leastthe overlapping fields of content that are specific to RIB and FIBoperability.

As illustrated in FIG. 13 , device 1300 (e.g., Leaf 104, Spine 102,etc.) can include Line Card Controller 1310, where the FIB resides andSUP Controller 1320, where the RIB resides. The routing protocols workson the SUP level (software level). The RIB is compiled to the Line CardController, which physically programs to the hardware of the device.Appliance 300 can determine the equivalence by extracting the FIB andRIB and convert them to a common format, as illustrated by table 1330.Table 1330 can show an IP address (e.g., 100.0.0.0), a mask (e.g., 32)and one or more next hops (e.g., NH1, NH2, etc.). Table 1330 can alsoinclude a “catch all” (e.g., *→DROP). For example, when an IP address isreceived that does not match an IP address/mask, the catch all will dropthe packets. The entries can be compared to determine there are noinconsistencies between the next hops.

FIG. 14 illustrates an example flowchart for an example networkassurance of a RIB-FIB equivalence of a network environment. The methodshown in FIG. 14 is provided by way of example, as there are a varietyof ways to carry out the method. Additionally, while the example methodis illustrated with a particular order of sequences, those of ordinaryskill in the art will appreciate that FIG. 14 and the sequences showntherein can be executed in any order that accomplishes the technicaladvantages of the present disclosure and can include fewer or moresequences than illustrated.

Each sequence shown in FIG. 14 represents one or more processes, methodsor subroutines, carried out in the example method. The sequences shownin FIG. 14 can be implemented in a network environment such as networkenvironment 100 shown in FIGS. 1A and 1B. The application can be presenton Appliance 300 and/or Checker 620, although the application is not solimited.

Method 1400 can begin at sequence 1410. At sequence 1410, an application(e.g., Appliance 300) can extract a RIB from the SUP Controller of adevice (e.g., Leaf 104, Spine 102, etc.). At sequence 1420, theapplication can extract a FIB from the Line Card Controller of thedevice. Both sequences 1410 and 1420 can entail polling one or more Leaf104 or Spine 102 for the content, or in other examples obtaining thecontent from the received Ci_Model 624 and Hi_Model for one or more Leaf104 or Spine 102.

At sequence 1430, the application can convert the extracted FIB and RIBto a common format (e.g., one of the FIB or RIB is converted into theformat of the other, or both are converted to the same new format). Atsequence 1440, the application can create a union of the converted RIBand FIB (e.g., a set union including lists all RIBs and FIBs withoutduplication), such as illustrated by Table 1330. Although shown as acollective table, the lists may be in distinct tables.

At sequence 1450, the application can identify a first entry in the RIBand determine if there is a matching IP address/mask in the FIB with amatching next hop. If such a match exists, then control passes tosequence 1470 to see if there is a next entry to consider. If so,control passes back to sequence 1450 to check the next entry (e.g., thenext entry in the RIB, and then turning to the entries in the FIBs).

For example with respect to Table 1330, the first entry RIB 1) has an IPaddress/mask 100.0.0.0/32 with a next hop of NH1. The same IPaddress/mask 100.0.0.0/32 with a next hop of NH1 is found in the FIB, atFIB 1). Since the RIB and FIB entries match, method 1400 can proceed tothe next entry sequence 1470. If there was no next entry then method1400 can end. In this example there is a next entry of RIB 2), and thuscontrol passes to sequence 1450 for the checking of the next entry ofRIB 2).

At sequence 1450, if the application does not identify a matching FIBentry for the RIB entry under inspection, then control passes sequence1460. At sequence 1460, the application can determine if the mismatch iscovered by another prefix, for example, via Longest Prefix Match. Whenthere is not another prefix and/or if the next hop does not match, anerror can be generated at 1480.

For example with respect to Table 1330, the entry RIB 2) 100.0.0.1/32does not have a matching FIB, and thus control passes to sequence 1460to identify another covering prefix, which in this case exists as FIB 2)100.0.0.*/30. If the next hops matched (between RIB 2 and FIB 2) controlwould pass to sequence 1450 for the next RIB entry. However, the nexthops of RIB 2) and FIB 2) do not match (NH1 v. NH2) and the error isnoted at sequence 1480 before control returns to sequence 1470 considerthe next entry.

In some cases, the mismatch can be a non-entry in one of the tables. Forexample, FIB 3) 200.0.0.0/30 does not have a match in the RIB table,except the “catch all” (e.g., *→Drop). Any incoming IP address 200.0.0.0will be dropped, even though the FIB table says route to NH1. Inresponse to this type of mismatch, an error can be generated.

In some cases, there can be entries in a RIB or FIB that will not beused. For example, RIB 5) will never be used since combined RIB 1) toRIB 4) cover all IPs in RIB 5).

When all RIBs are examined, the process continues for the list of FIBs,and particularly those FIBs that were not accounted for by the initialRIB check. The process could also be reversed, in that the list of FIBscould be checked before RIBs. Or the system could switch between lists.The application is not limited to the manner in which the list of RIBsand FIBs is processed.

Validation of L3out (Internal-External Forwarding)

As shown above in FIG. 7 , internal subnets (e.g., 100.2.*) can beleaked to external networks (e.g., Network Device 114) and externalnetworks (e.g., 175.*) can be leaked to the Fabric. In order forcommunication to properly occur between the public subnets in the Fabric120 and the external devices, Layer 3 out (“L3out”) must be properlyconfigured. Appliance 300 can determine that the L3out is properlyconfigured.

The L3out interfaces model can include, at a minimum, leaf (e.g.,border), port and network (e.g., 40.1.2.*). Border leafs can have L3outconfigured to access an outside device/network (e.g., Network Device114). To deploy an L3out interface the leaf, port and network can beconfigured. For example, an operator can program Leaf N, port eth1/10and network 40.1.2.* into L_Model 270A, for which in a manner discussedabove Leaf N generates the corresponding CN_Model and HN_Model. Onceconfigured, the link between Leaf N and Network Device 114 is importantin transferring data external the fabric. In this case, network 40.1.2.*would own the link. The 100.2.* subnet (as shown in LPM Table 748) canbe leaked to Network Device 114 (through L3out interface and link40.1.2.*). The subnet is leaked through routing protocols, such as, BGP,OSPF, EIGRP, etc. For example, in the routing protocols there are twopeers: peer1 is the border leaf (e.g., Leaf N) and peer2 is the externaldevice (e.g., Network Device 114). The two peers will exchange theirroutes with each other. For example, peer1 can tell peer2 that it haspublic networks (e.g., 100.2.*) to leak outside the fabric and peer 2can tell peer1 that it has external routes (e.g., 175.*) to be leakedinside the fabric.

L3out loopback is an interface for configuring networking protocols(e.g., BGP). Static routes can be programmed by the Fabric 120. Forexample, Network Device 114 can have network 200.* that has not beenleaked to the Fabric 120. In these cases, the L_Model can directlyprogram the 200.* network for communicating with an external router(e.g., Network Device 114). EPG (Instp) can be used to control import ofexternal routes. For example, Network Device 114 can have several routes(e.g., 100.*, 300.*, 400.*, etc.) leaked to Fabric 120. These networkscan be tagged with EPGs. For example, 100.* can be tagged with EPG1(e.g., Logical Group 1), 300.* can be tagged with EPG2 (e.g., LogicalGroup 2) and 400.* can be tagged with EPG3 (e.g., Logical Group 3). Assuch, whenever an incoming packet comes into Fabric 120 from thesenetworks, the packet will be tagged with the associated EPG.

Appliance 300 can perform checks on the L3out through an L-C Modelcheck. For example, Appliance 300 can verify the interface, loopback,and static routes are programmed on the border leafs (e.g., present inthe LPM Table). Appliance 300 can also verify one or more EPGs of L3outare correct. Appliance 300 can verify, for each network, the next hop isproper.

FIG. 15 illustrates an example flowchart for an example networkassurance of L3out configuration of a network environment. The methodshown in FIG. 15 is provided by way of example, as there are a varietyof ways to carry out the method. Additionally, while the example methodis illustrated with a particular order of sequences, those of ordinaryskill in the art will appreciate that FIG. 15 and the sequences showntherein can be executed in any order that accomplishes the technicaladvantages of the present disclosure and can include fewer or moresequences than illustrated.

Each sequence shown in FIG. 15 represents one or more processes, methodsor subroutines, carried out in the example method. The sequences shownin FIG. 15 can be implemented in a network environment such as networkenvironment 700 shown in FIG. 7 . The application can be present onAppliance 300 and/or Checker 620, although the application is not solimited.

Method 1500 can begin at sequence 1510. At sequence 1510 the variousmodels (e.g., L_Model, C_Model, and/or H_Model) to be checked arereceived. For example, one or more Leafs 104, Spines 102 and Controllers116 in Fabric 120 can send to the application (e.g., Assurance Appliance300) L_Model 270A/B, Ci_Model 274, and Ci_Model 276. Those receivedmodels can include information content that can be subject to thevalidation process. In some examples, the information relevant to anL3out assurance check can include interface (e.g., leaf, port, network),loopback (e.g., protocol), static routes, and EPG.

At sequence 1520, the application can validate networks configured tocommunicate through L3out are in the LPM Table (e.g., L3 Table). The LPMTable (e.g., C_Model) can contain configuration of the devices in thenetwork (e.g., Leafs 104, Spines 102, etc.). The L_Models and C_Modelscan be converted to a common format as discussed above, and the relevantcontent of overlapping fields can be compared. For example, the L3outinterface assurance considers the fields of leaf, port and network.Appliance 300 can validate that the leaf from the configuration has theport and network deployed as shown in the configuration (e.g., L_Model).Appliance 300 can perform similar validation for the loopback(consideration of fields of leaf and network) and static routes(consideration of the fields of leaf, network and next hop). EPG can bevalidated by the application as well via an L-C check.

At sequence 1530, the application can validate the next hops via C_Modelconsistency check. For example, validating the LPM Tables at each leafincludes next hop entries to communicate with the border leaf (e.g., forexternal communication). As shown in FIG. 7 , Leaf 2 can include an LPMTable 742. LPM Table 742 can include subnets and next hops. Aspreviously discussed, subnet 100.2.* has been leaked from Leaf N toNetwork Device 114 and when Leaf 2 wants to communication with theexternal network, data is routed through Leaf N (or another border leafconnected to Network Device 114). Appliance 300 can validate the nexthop routes in the LPM tables of the leafs and spines are correct.

At sequence 1540, one or more events can be generated. For example, whenthe validations are consistent (e.g., consistent with expectations ofthe validation) the application can generate an informational event. Inother examples, when one or more of the validations are not consistent(e.g., inconsistent with expectations of the validation), configurationsare not on the correct leafs (according the L_Model) or when there aremissing or extra information the application can generate an errorevent. The severity of the error event can at least be determined basedon which validation is inconsistent and which device had theinconsistency (e.g., production device, test device, access point,border leaf, etc.).

Validation of BD-L3out Configuration (Internal-External Forwarding)

BD-L3out association enables internal networks to be leaked external tothe fabric, that is, the external router should know about internalnetworks and where data should be routed for accessing the internalnetworks (e.g., internal Fabric 120). As shown in FIG. 7 , Leaf 2 has apublic network 100.2.*. For network 100.2.* to be accessed from outsidethe fabric (e.g., by network device 114), the BD-L3out association canbe configured. As shown in FIG. 16 , the BD-L3out association caninclude, BD subnet is marked public (e.g., 100.2.*), BD is associatedwith L3out (e.g., 1650) and a security contract exists between the EPGof BD and EPG of L3out (e.g., Contract 1). When BD subnet is markedpublic and BD is associated with L3out, routing can work properlyexternal to the fabric.

Appliance 300 can validate these BD-L3out configuration conditions. Inparticular, Appliance 300 can confirm that the models reflect that theinternal networks have in fact been leaked. This specific check uses aseries of L_Model consistency checks (or a single check that considersmultiple fields of information).

FIG. 17 illustrates an example flowchart for an example networkassurance of BD-L3out association configuration of a networkenvironment. The method shown in FIG. 17 is provided by way of example,as there are a variety of ways to carry out the method. Additionally,while the example method is illustrated with a particular order ofsequences, those of ordinary skill in the art will appreciate that FIG.17 and the sequences shown therein can be executed in any order thataccomplishes the technical advantages of the present disclosure and caninclude fewer or more sequences than illustrated.

Each sequence shown in FIG. 17 represents one or more processes, methodsor subroutines, carried out in the example method. The sequences shownin FIG. 17 can be implemented in a network environment such as networkenvironment 700 shown in FIG. 7 and network diagram 1600 shown in FIG.16 . The application can be present on Appliance 300 and/or Checker 620,although the application is not so limited.

At sequence 1710 the various models (e.g., L_Model, C_Model, and/orH_Model) to be checked are received. For example, each Controller 116 inthe Fabric 120 can send to the application (e.g., Assurance Appliance300 and/or Checker 620) one or more L_Models 270A/B, and one or moreLeafs 104 and Spines 102 can send corresponding Ci_Model 274, andCi_Model 276. In another example, some but not all of the L_Model,C_Model, and/or H_Model configurations are provided and/or received.Those received models can include information content, at least some ofwhich will be subject to the validation process. In some examples, theinformation can include L3out configuration data and BD configurationdata.

At sequence 1720, the application via an L_Model consistency checkidentifies whether any BD subnets are marked public. In some examples,BD subnets in the fabric are configured to be private, shared, etc. Inorder to be marked public, the operator must specify the subnet aspublic in L_Model 270A and push the configuration to the Controller 116.The application can verify via an L-Model consistency check whether anyBD subnet in the received L_Model 622 is marked public. If not, thenthis check ends at sequence 1760. If one or more of the BD subnets inthe received L_Model 622 are marked as public, then the processcontinues at 1730 for each subnet identified as public; the process willbe described with respect to a single subnet so identified, although itis to be understood that the process would also apply to multipleidentified subnets.

At sequence 1730, the application via an L_Model consistency check canvalidate the public BD is associated with an L3out. The BD is associatedwith L3out through the BD configuration. The BD configuration can beprogrammed by the operator in L_Model 270A at the Controller 116 (e.g.,Tenant Configuration 740), and which can subsequently arrive at checker620 via sequence 1710 above. An example of a BD-L3out association 1650is shown in FIG. 16 . The application can verify the association via anL-Model consistency check.

At sequence 1740, the application via an L_Model consistency check canvalidate a contract between the EPGs (e.g., Logical Groups). As shownabove, the L3out can have an EPG (e.g., EPG2) and BD can have an EPG(e.g., EPG1). When a contract (e.g., Contract 1) has been formed betweenthe EPGs, the associated endpoints can then communicate. Appliance 300can validate that a security contract is present between the EPG underL3out and the EPG under BD.

At sequence 1750, one or more events can be generated. For example, whenthe validations are consistent (e.g., consistent with expectations ofthe validation) the application can generate an information event. Inother examples, when one or more of the validations are not consistent(e.g., inconsistent with expectations of the validation), configurationsare not on the correct leafs (according the L_Model) or when there ismissing or extra information the application can generate an errorevent. The severity of the error event can at least be determined basedon which validation is inconsistent and which device had theinconsistency (e.g., production device, test device, access point,border leaf, etc.).

Proper Propagated Learned (Imported) Routes

Learned Routes (e.g., imported routes) are external networks that areleaked (i.e., imported) inside the Fabric 120. The external networks(e.g., to be leaked) can be networks that an external device (e.g.,Network Device 114) wants to share with the Fabric 120 via a border leaf(e.g., Leaf N). The Fabric 120 can in theory import all networksprovided by the external router or only specific external networks. Thedecisions can be based on resources of the network devices, for examplethe LPM table. When all external routes are imported, for examplethousands of networks, expensive resources are utilized by storing theselearned routes at each network device (e.g., Leafs 104, Spines 102,etc.). The learned routes populate the LPM Tables stored in the TCAM ofeach network device. Overpopulating these resources can affect theefficiency and overall operation of the fabric. The border leaf (e.g.,Leaf N) can decide (e.g., based on operator intent) which externalnetworks to import.

The deployment of the instruction to permit a leak of a particularexternal network is programmed by the operator into the L_Model 270A,and can identify for a particular border Leaf 104 (e.g., Leaf N) whatnetwork(s) can be leak into Fabric 120. The particular border Leaf 104(e.g., Leaf N) can store the permissible network(s) in EPGs defined inthe L3out table (e.g., table 1806 for Leaf N) within the Ci_Model 274.

When an external device (e.g., Network Device 114) wants to share anexternal network with the Fabric 102 via a border leaf (e.g., Leaf N),the border leaf (e.g., Leaf N) initially determines from its L3out tablewhether import of that network is permissible. If permissible, theborder leaf (e.g., Leaf N) sets that external device as the next hop inthe LPM table (e.g., 748 for Leaf N). The L3out of the border Leaf(e.g., Leaf N) is under a particular VRF, and as the “source” of theleak the border Leaf propagates (e.g., leaks) that learned route toother Leafs 104 (e.g., student leafs) which have an L3out or BD that isunder that same particular VRF. Those “student” Leafs 104 will store thelearned route in their LPM table (e.g., 1802 for Leaf 1 and 742 for Leaf2) and set the next hop to the source Leaf (e.g. Leaf N). By virtue ofthe source Leaf and the student Leaf having L3out and/or BD under thesame VRF, these source and student Leafs 104 define a “VRF Leaf Group.”

Non-limiting examples of learned routes are illustrated in Fabric 120 ofFIG. 7 . L_Model 270A/B contains instructions for Leaf N that it canimport from external network 175.*. Leaf N includes this in its CN_model272 as an L3out in LPM table 1806. An external Network Device 114 wantsto export networks to Fabric 120 through Border Leaf N, for example,external subnets 175.*, 185.*, 195.* as listed in routing and forwardingtable 744 of Network Device 114. Border Leaf N can import network 175.*(e.g., EPG (175.*) because it is listed in table 1806, but cannot importnetworks 185.* and 195.* because they are not present in table 1806.Border Leaf N updates table 748 along with the next hop indictor ofnetwork device 114 (175.*→Device 114). Border Leaf N can then spreadsthe leak to the remainder of the VRF Leaf Group, which in this caseincludes Leaf 1 and Leaf 2. Leaf 1 and Leaf 2 can update their LPM table(e.g., 1802, 742) at the Ci_Model 274 level to reflect the import of175.* and set Leaf N as the corresponding next hop to reflect it is thesource Leaf (e.g., 175.*→Leaf N).

In some examples, multiple border Leafs import a network from the sameexternal device. In FIG. 7 , Leaf 3 is also a border leaf as shown bydashed line 1804 to external Network Device 114, and as such alsoimports network 175.* and propagates the imported networks to Leafs 1and 2. The LPM tables (e.g., 1802, 742) reflect that the next hop can beeither Leaf 3 or Leaf N (e.g., 175.*→Leaf 3, N).

In some examples, the LPM tables may have errors. For example, the LPMTable of the devices (e.g., 1802, 742, etc.) can include equivalentlearned routes as the LPM Table of the border leaf (e.g., 748). Turningto FIG. 7 , the LPM Table of Leaf 2 (e.g., 742) shows an equivalentlearned route (e.g., 175.*) as in Border Leaf N, while the LPM Table ofLeaf 1 (e.g., 1802) shows an extra learned route (e.g., 185.*) not inborder Leaf N. Network 185.* was not authorized for importation byL_Model 270A. The presence of learned route 185.* is thus a discrepancyand can generate an error.

FIG. 18 illustrates an example flowchart for an example networkassurance of learned routes of a network environment. The method shownin FIG. 18 is provided by way of example, as there are a variety of waysto carry out the method. Additionally, while the example method isillustrated with a particular order of sequences, those of ordinaryskill in the art will appreciate that FIG. 18 and the sequences showntherein can be executed in any order that accomplishes the technicaladvantages of the present disclosure and can include fewer or moresequences than illustrated.

Each sequence shown in FIG. 18 represents one or more processes, methodsor subroutines, carried out in the example method. The sequences shownin FIG. 18 can be implemented in a network environment such as networkenvironment 700 shown in FIG. 7 . The application can be present onAppliance 300 and/or Checker 620, although the application is not solimited.

Method 1800 can begin at sequence 1810. At sequence 1810, an application(e.g., Assurance Appliance 300) can receive various models (e.g.,L_Model, C_Model, and/or H_Model) to be checked. For example, eachController 116 in the Fabric 120 can send to the application (e.g.,Assurance Appliance 300 and/or Checker 620) one or more L_Models 270A/B,and each Leaf 104 and Spine 102 sends its corresponding Ci_Model 274,and Ci_Model 276. In another example, some but not all of the L_Model,C_Model, and/or H_Model configurations are provided and/or received.Those received models will include information content, at least some ofwhich will be subject to the validation process. For example, eachborder leaf (e.g., Leaf 3, Leaf N, etc.) in the Fabric 120 can send tothe application, information associated with software model (C_Model).In some examples, the information can be the LPM Table (e.g., L3 Table)and L3out configuration.

Learned routes can be present in the LPM Table within the Ci_Model 274of border nodes (e.g., border Leaf) and the EPG of L3out in the L_Model270A. At sequence 1820, the application examines the Ci_Model 624 forone or more border Leafs along with the L_Model 622 to confirm that thelearned routes are consistent with each other. For example, when the EPGof L3out includes the imported route 175.*, the LPM Table should include175.* as an imported route (and no other learned routes, unless importedfrom a different border leaf or a different L3out). In response todiscrepancies of the learned routes between EPG of L3out and the LPMTable an error event can be generated (e.g., at sequence 1860). Inresponse to no discrepancies, an informational event can be generated.

Sequence 1820 may require a common format step for Ci_Model 624 andL_Model 622, although the subject matter of interest may be extract fromoverlapping fields of the models the common format as well as fromfields in the models that do not overlap.

At sequence 1830, the application can validate the learned routes arepresent on the “source” are “student” leafs (e.g., where the VRF for theEPG/Logical Groups are present). The application thus checks theCi_Model 624 configuration for all of Leafs within a common VRF LeafGroup to confirm that the learned routes are consistent across thegroup. This is can be a C-Model level consistency check per Ci_Model andacross multiple Ci_Models. With respect to FIG. 7 , this check wouldconfirm that all listings of 175.* are proper in those Leafs; it wouldalso identify that network 185.* is improperly listed in 1802 andtrigger a corresponding error event.

At sequence 1840, the application can validate the next hops for theimported routes. For a learned route in a source Leaf, this is aCi_Model level consistency check to validate that the next hop isdirected to the external device. By way of non-limiting example, in FIG.7 the CN_Model for Leaf N is checked to confirm that the next hop fornetwork 175 is the external device that requested import (e.g.,175.*→Device 114). In the example of a learned route in a student leaf,this is a Ci_Model level consistency check to validate that the next hopis directed to the source Leaf(s). By way of non-limiting example, inFIG. 7 the Ci_Model for Leafs 1 and 2 are checked to confirm that thelearned routes (e.g., 175.*) are directed to the border leafs (e.g.,Leaf 3 and N). When there is more than one border leaf and two next hops(e.g., 175, e.g., 175.*→Leaf 3, N) the application can validate bothnext hops (for network redundancy this is common).

When routing between two next hops, the leaf (e.g., Leaf 1) can computea hash based on flow (e.g., source IP, source port, destination IP,destination port, etc.). The computed hash will be modulo with thenumber of border leafs. In this example, there are two border leafs sothe computed hash modulo 2 (e.g., hash % 2). The routing would then besprayed among the modulo output.

At sequence 1850, one or more events can be generated. For example, whenthe validations are consistent (e.g., consistent with expectations ofthe validation) the application can generate an information event. Inother examples, when one or more of the validations are not consistent(e.g., inconsistent with expectations of the validation), configurationsare not on the correct leafs (according the L_Model) or when there aremissing or extra information the application can generate an errorevent. The severity of the error event can at least be determined basedon which validation is inconsistent and which device had theinconsistency (e.g., production device, test device, access point,border leaf, etc.).

Detection of Overlapping Subnets

An overlapping subnet in a network (e.g., Fabric 120) can causemisrouting and lost packets during forwarding (e.g., using LPM). In afabric there can be multiple networks in each VRF, for example, BDsubnets; EPG subnets (function same as BD subnets); L3out interface;L3out loopback; L3out static routes; L3out loopbacks; learned routes;etc. These multiple networks can be determined by operator intent orimported (e.g., learned). Each of the networks (e.g., subnets) can havean IP address and a mask (e.g., IPv4, IPv6, etc.). The mask determineshow much of the IP address is used in forwarding data (e.g., LPM). Forexample, an IP/mask of 200.1.1.1/16 would be used as 200.1.* because thelast 16 bits (in an IPv4 address) are masked.

FIG. 19 illustrates network diagram 1900 of potential subnet overlapsand exceptions. Generally, subnets in a fabric can be spread out acrossthe possible subnets. However, operator intent (e.g., misintent) andmasks can create unintended overlap. For example, BD1 subnets and BD2subnets can overlap in that BD1 and BD2 can be configured with at leastone identical subnets (e.g., both are configured with a subnet of100.1.1.1). In this example, there is an overlap and an event isgenerated. In other examples, this overlap can be constrained by themask. For example, BD1 can have a mask of 32 (e.g., 100.1.1.1/32) andBD2 can have a mask of 24 (e.g., 100.1.1.1/24). In this example, whenforwarding data BD1 would have subnet 100.1.1.1 and BD2 would havesubnet 100.1.1.*. As such, anything other than 100.1.1.1 would beforwarded to BD2.

While the above overlap is constrained by the mask, a complete overlapis still possible as shown by BD3 subnets and BD4 subnets. For example,BD3 subnets can include 200.1.1.0/16 and BD4 subnets can include200.1.1.1/24. In this example, while the subnets are different, the maskwould overlap them, BD3 subnet would be 200.1.* and BD4 subnet would be200.1.1.*. As such, any forwarding to 200.1.1.0 (or 200.1.1.*) would goto BD4 subnet even though the intent could be to send to BD3 subnet(because LPM). This overlapping would generate an event. One exceptionto this would be if BD3 and BD4 were the same BD (e.g., forwarding wouldbe to the same BD so no clash).

In some situations, the EPGs to which there is a security contract canbe in different VRFs, as shown in FIG. 12 (e.g., dashed line from BD2 toVRF2). For example, BD2 can be bound to VRF2 and BD1 can be bound toVRF1. When Security Contract 2 is formed between EPG3 (bound to BD2) andEPG1 (bound to BD1) the subnets from BD2 (e.g., 300.* and 400.*) areleaked to VRF1 and the subnets from BD1 (e.g., 100.* and 200.*) areleaked to VRF2. Once, the subnets are leaked, VRF1 cannot use subnets300.* and 400.* independently and VRF2 cannot use subnets 100.* and200.* independently without their being a clash of subnets. Appliance300 is configured to verify and check there are no clashes betweensubnets when contracts are formed between EPGs.

There are some exceptions to the overlapping which are shown by theLearned Routes and BD5 subnets. In this example, the Learned Routes canhave a subnet of 200.* (e.g., mask 8) and BD5 subnet can have a subnetof 200.1.1.*. The BD5 subnet is owned by the fabric (e.g., internal) andthe Learned Routes is owned external. In this example, the overlap doesnot matter as when using LPM because anything with a 200.1.1.XXX will beforwarded to BD5, and everything else will be forwarded external. Assuch, internal routing is given preference over external and there willbe no misrouting of data. However, when the situations are reversed (BD5subnet being 200.* and external being 200.1.1.*), the internal networkis not provided preference and internal data could be sent outside thefabric, and an error can be generated.

FIG. 20 illustrates an example flowchart for an example networkassurance of learned routes of a network environment. The method shownin FIG. 20 is provided by way of example, as there are a variety of waysto carry out the method. Additionally, while the example method isillustrated with a particular order of sequences, those of ordinaryskill in the art will appreciate that FIG. 20 and the sequences showntherein can be executed in any order that accomplishes the technicaladvantages of the present disclosure and can include fewer or moresequences than illustrated.

Each sequence shown in FIG. 20 represents one or more processes, methodsor subroutines, carried out in the example method. The sequences shownin FIG. 20 can be implemented in a network environment such as networkenvironment 700 shown in FIG. 7 and network diagram 1900 shown in FIG.19 . The application can be present on Appliance 300 and/or Checker 620,although the application is not so limited.

Method 2000 can begin at sequence 2010. At sequence 2010 an application(e.g., Assurance Appliance 300) receives various models (e.g., L_Model,C_Model, and/or H_Model) to be checked. For example, one or moreControllers 116 in the Fabric 120 can send to the application (e.g.,Assurance Appliance 300 and/or Checker 620) one or more L_Models 270A/B,and one or more Leafs 104 and Spines 102 can send corresponding Ci_Model274, and Ci_Model 276. In another example, some but not all of theL_Model, C_Model, and/or H_Model configurations are provided and/orreceived. Those received models will include information content, atleast some of which will be subject to the validation process. In someexamples, the information can include the multiple networks in each VRF(e.g., from the LPM table, tenant config, etc.).

At sequence 2020, the application can determine impermissible overlapbetween the received networks for each VRF (e.g., in the L_Modelsreceived). Subnets can be found in the L_Model 270A/B, and learnedroutes can be found in individual Ci_Model 274 configuration for eachVRF. All of these subnets, including learned routes, can be examined todirect an overlap. An overlap for which no exception exists constitutesa discrepancy.

At sequence 2030, one or more events can be generated. For example, whenthe validations are consistent (e.g., consistent with expectations ofthe validation) the application can generate an information event. Inother examples, when one or more of the validations are not consistent(e.g., inconsistent with expectations of the validation), configurationsare not on the correct leafs (according the L_Model) or when there aremissing or extra information the application can generate an errorevent. The severity of the error event can at least be determined basedon which validation is inconsistent and which device had theinconsistency (e.g., production device, test device, access point,border leaf, etc.).

The foregoing embodiments have been described with reference to aLeaf/Spine topology. However, the application is not so limited, and themethodologies can be applied to other topologies.

The disclosure now turns to FIGS. 21 and 22 , which illustrate examplenetwork devices and computing devices, such as switches, routers, loadbalancers, client devices, and so forth.

FIG. 21 illustrates an example network device 2100 suitable forperforming switching, routing, load balancing, and other networkingoperations. Network device 2100 includes a central processing unit (CPU)2104, interfaces 2102, and a bus 2110 (e.g., a PCI bus). When actingunder the control of appropriate software or firmware, the CPU 2104 isresponsible for executing packet management, error detection, and/orrouting functions. The CPU 2104 can accomplish all these functions underthe control of software including an operating system and anyappropriate applications software. CPU 2104 may include one or moreprocessors 2108, such as a processor from the INTEL X86 family ofmicroprocessors. In some cases, processor 2108 can be specially designedhardware for controlling the operations of network device 2100. In somecases, a memory 2106 (e.g., non-volatile RAM, ROM, etc.) also forms partof CPU 2104. However, there are many different ways in which memorycould be coupled to the system.

The interfaces 2102 are typically provided as modular interface cards(sometimes referred to as “line cards”). Generally, they control thesending and receiving of data packets over the network and sometimessupport other peripherals used with the network device 2100. Among theinterfaces that may be provided are Ethernet interfaces, frame relayinterfaces, cable interfaces, DSL interfaces, token ring interfaces, andthe like. In addition, various very high-speed interfaces may beprovided such as fast token ring interfaces, wireless interfaces,Ethernet interfaces, Gigabit Ethernet interfaces, ATM interfaces, HSSIinterfaces, POS interfaces, FDDI interfaces, WIFI interfaces, 3G/4G/5Gcellular interfaces, CAN BUS, LoRA, and the like. Generally, theseinterfaces may include ports appropriate for communication with theappropriate media. In some cases, they may also include an independentprocessor and, in some instances, volatile RAM. The independentprocessors may control such communications intensive tasks as packetswitching, media control, signal processing, crypto processing, andmanagement. By providing separate processors for the communicationsintensive tasks, these interfaces allow the master microprocessor 604 toefficiently perform routing computations, network diagnostics, securityfunctions, etc.

Although the system shown in FIG. 21 is one specific network device ofthe present invention, it is by no means the only network devicearchitecture on which the present invention can be implemented. Forexample, an architecture having a single processor that handlescommunications as well as routing computations, etc., is often used.Further, other types of interfaces and media could also be used with thenetwork device 2100.

Regardless of the network device's configuration, it may employ one ormore memories or memory modules (including memory 2106) configured tostore program instructions for the general-purpose network operationsand mechanisms for roaming, route optimization and routing functionsdescribed herein. The program instructions may control the operation ofan operating system and/or one or more applications, for example. Thememory or memories may also be configured to store tables such asmobility binding, registration, and association tables, etc. Memory 2106could also hold various software containers and virtualized executionenvironments and data.

The network device 2100 can also include an application-specificintegrated circuit (ASIC), which can be configured to perform routingand/or switching operations. The ASIC can communicate with othercomponents in the network device 2100 via the bus 2110, to exchange dataand signals and coordinate various types of operations by the networkdevice 2100, such as routing, switching, and/or data storage operations,for example.

FIG. 22 illustrates a computing system architecture 2200 wherein thecomponents of the system are in electrical communication with each otherusing a connection 2205, such as a bus. Exemplary system 2200 includes aprocessing unit (CPU or processor) 2210 and a system connection 2205that couples various system components including the system memory 2215,such as read only memory (ROM) 2220 and random access memory (RAM) 2225,to the processor 2210. The system 2200 can include a cache of high-speedmemory connected directly with, in close proximity to, or integrated aspart of the processor 2210. The system 2200 can copy data from thememory 2215 and/or the storage device 2230 to the cache 2212 for quickaccess by the processor 2210. In this way, the cache can provide aperformance boost that avoids processor 2210 delays while waiting fordata. These and other modules can control or be configured to controlthe processor 2210 to perform various actions. Other system memory 2215may be available for use as well. The memory 2215 can include multipledifferent types of memory with different performance characteristics.The processor 2210 can include any general purpose processor and ahardware or software service, such as service 1 2232, service 2 2234,and service 3 2236 stored in storage device 2230, configured to controlthe processor 2210 as well as a special-purpose processor where softwareinstructions are incorporated into the actual processor design. Theprocessor 2210 may be a completely self-contained computing system,containing multiple cores or processors, a bus, memory controller,cache, etc. A multi-core processor may be symmetric or asymmetric.

To enable user interaction with the computing device 2200, an inputdevice 2245 can represent any number of input mechanisms, such as amicrophone for speech, a touch-sensitive screen for gesture or graphicalinput, keyboard, mouse, motion input, speech and so forth. An outputdevice 2235 can also be one or more of a number of output mechanismsknown to those of skill in the art. In some instances, multimodalsystems can enable a user to provide multiple types of input tocommunicate with the computing device 2200. The communications interface2240 can generally govern and manage the user input and system output.There is no restriction on operating on any particular hardwarearrangement and therefore the basic features here may easily besubstituted for improved hardware or firmware arrangements as they aredeveloped.

Storage device 2230 is a non-volatile memory and can be a hard disk orother types of computer readable media which can store data that areaccessible by a computer, such as magnetic cassettes, flash memorycards, solid state memory devices, digital versatile disks, cartridges,random access memories (RAMs) 2225, read only memory (ROM) 2220, andhybrids thereof.

The storage device 2230 can include services 2232, 2234, 2236 forcontrolling the processor 2210. Other hardware or software modules arecontemplated. The storage device 2230 can be connected to the systemconnection 2205. In one aspect, a hardware module that performs aparticular function can include the software component stored in acomputer-readable medium in connection with the necessary hardwarecomponents, such as the processor 2210, connection 2205, output device2235, and so forth, to carry out the function.

For clarity of explanation, in some instances the present technology maybe presented as including individual functional sequences includingfunctional sequences comprising devices, device components, steps orroutines in a method embodied in software, or combinations of hardwareand software.

In some embodiments the computer-readable storage devices, mediums, andmemories can include a cable or wireless signal containing a bit streamand the like. However, when mentioned, non-transitory computer-readablestorage media expressly exclude media such as energy, carrier signals,electromagnetic waves, and signals per se.

Methods according to the above-described examples can be implementedusing computer-executable instructions that are stored or otherwiseavailable from computer readable media. Such instructions can comprise,for example, instructions and data which cause or otherwise configure ageneral purpose computer, special purpose computer, or special purposeprocessing device to perform a certain function or group of functions.Portions of computer resources used can be accessible over a network.The computer executable instructions may be, for example, binaries,intermediate format instructions such as assembly language, firmware, orsource code. Examples of computer-readable media that may be used tostore instructions, information used, and/or information created duringmethods according to described examples include magnetic or opticaldisks, flash memory, USB devices provided with non-volatile memory,networked storage devices, and so on.

Devices implementing methods according to these disclosures can comprisehardware, firmware and/or software, and can take any of a variety ofform factors. Typical examples of such form factors include laptops,smart phones, small form factor personal computers, personal digitalassistants, rackmount devices, standalone devices, and so on.Functionality described herein also can be embodied in peripherals oradd-in cards. Such functionality can also be implemented on a circuitboard among different chips or different processes executing in a singledevice, by way of further example.

The instructions, media for conveying such instructions, computingresources for executing them, and other structures for supporting suchcomputing resources are means for providing the functions described inthese disclosures.

Although a variety of examples and other information was used to explainaspects within the scope of the appended claims, no limitation of theclaims should be implied based on particular features or arrangements insuch examples, as one of ordinary skill would be able to use theseexamples to derive a wide variety of implementations. Further andalthough some subject matter may have been described in languagespecific to examples of structural features and/or method steps, it isto be understood that the subject matter defined in the appended claimsis not necessarily limited to these described features or acts. Forexample, such functionality can be distributed differently or performedin components other than those identified herein. Rather, the describedfeatures and steps are disclosed as examples of components of systemsand methods within the scope of the appended claims.

Claim language reciting “at least one of” refers to at least one of aset and indicates that one member of the set or multiple members of theset satisfy the claim. For example, claim language reciting “at leastone of A and B” means A, B, or A and B.

What is claimed is:
 1. A system to perform a network assurance check ofa configuration in a fabric, the system comprising: at least one memoryconfigured to store data; and at least one processor operable to executeinstructions associated with the data which, when executed by the atleast one processor, cause the at least one processor to: receive aglobal logical model, the global logical model including a plurality oflogical network segments and a plurality of endpoints associated with atleast one logical group; create a first device-specific configurationimplementing a portion of the global logical model; send the firstdevice-specific configuration over a network connection to a firstswitch; receive, via an application programming interface (API),information about a plurality of endpoints connected to a network, theinformation for each endpoint including at least one attribute, at leastone logical group association, and at least one rule specifying acommunication restriction applicable to the endpoint; determine whetherthere is a configuration conflict between the received information aboutthe plurality of endpoints and either or both of a) the global logicalmodel or b) the first device-specific configuration; based on adetermination that there is a conflict, generate a seconddevice-specific configuration; and send the second device-specificconfiguration over the network connection to a first switch.
 2. Thesystem of claim 1, wherein the device-specific configuration includes avirtual routing and forwarding (VRF) container.
 3. The system of claim2, wherein the a VRF associated with the VRF container has under it atleast one bridge domain (BD) and at least one associated endpoint group(EPG), the at least one BD including at least one subnet.
 4. The systemof claim 2, wherein the second device-specific configuration includes anupdate for the VRF container.
 5. The system of claim 1, wherein thefirst switch is configured to receive the first device-specificconfiguration and update its hardware-level configuration to conform tothe first device-level configuration.
 6. The system of claim 5, whereinthe first switch is configured to receive the second device-specificconfiguration and update its hardware-level configuration to conform tothe second device-level configuration.
 7. At least one non-transitorycomputer-readable medium having stored thereon instructions which, whenexecuted by one or more processors, cause the one or more processors to:receive a global logical model, the global logical model including aplurality of logical network segments and a plurality of endpointsassociated with at least one logical group; create a firstdevice-specific configuration implementing a portion of the globallogical model; send the first device-specific configuration over anetwork connection to a first switch; receive, via an applicationprogramming interface (API), information about a plurality of endpointsconnected to a network, the information for each endpoint including atleast one attribute, at least one logical group association, and atleast one rule specifying a communication restriction applicable to theendpoint; determine whether there is a configuration conflict betweenthe received information about the plurality of endpoints and either orboth of a) the global logical model or b) the first device-specificconfiguration; based on a determination that there is a conflict,generate a second device-specific configuration; and send the seconddevice-specific configuration over the network connection to a firstswitch.
 8. The at least one non-transitory computer-readable medium ofclaim 7, wherein the device-specific configuration includes a virtualrouting and forwarding (VRF) container.
 9. The at least onenon-transitory computer-readable medium of claim 8, wherein the a VRFassociated with the VRF container has under it at least one bridgedomain (BD) and at least one associated endpoint group (EPG), the atleast one BD including at least one subnet.
 10. The at least onenon-transitory computer-readable medium of claim 8, wherein the seconddevice-specific configuration includes an update for the VRF container.11. The at least one non-transitory computer-readable medium of claim 7,wherein the first switch is configured to receive the firstdevice-specific configuration and update its hardware-levelconfiguration to conform to the first device-level configuration. 12.The at least one non-transitory computer-readable medium of claim 11,wherein the first switch is configured to receive the seconddevice-specific configuration and update its hardware-levelconfiguration to conform to the second device-level configuration. 13.The at least one non-transitory computer-readable medium of claim 7,wherein determining whether there is a configuration conflict comprisesidentifying one or more policy violations across the network.
 14. Amethod comprising: receiving a global logical model, the global logicalmodel including a plurality of logical network segments and a pluralityof endpoints associated with at least one logical group; creating afirst device-specific configuration implementing a portion of the globallogical model; sending the first device-specific configuration over anetwork connection to a first switch; receiving, via an applicationprogramming interface (API), information about a plurality of endpointsconnected to a network, the information for each endpoint including atleast one attribute, at least one logical group association, and atleast one rule specifying a communication restriction applicable to theendpoint; determining whether there is a configuration conflict betweenthe received information about the plurality of endpoints and either orboth of a) the global logical model or b) the first device-specificconfiguration; based on a determination that there is a conflict,generating a second device-specific configuration; and sending thesecond device-specific configuration over the network connection to afirst switch.
 15. The method of claim 14, wherein the device-specificconfiguration includes a virtual routing and forwarding (VRF) container.16. The method of claim 15, wherein a VRF associated with the VRFcontainer has under it at least one bridge domain (BD) and at least oneassociated endpoint group (EPG), the at least one BD including at leastone subnet.
 17. The method of claim 15, wherein the seconddevice-specific configuration includes an update for the VRF container.18. The method of claim 14, wherein the first switch is configured toreceive the first device-specific configuration and update itshardware-level configuration to conform to the first device-levelconfiguration.
 19. The method of claim 18, wherein the first switch isconfigured to receive the second device-specific configuration andupdate its hardware-level configuration to conform to the seconddevice-level configuration.
 20. The method of claim 14, whereindetermining whether there is a configuration conflict comprisesidentifying one or more policy violations across the network.